Huihui-Qwen3-VL-4B-Instruct-abliterated — ComfyUI Edition

This repo packages the huihui-ai/Huihui-Qwen3-VL-4B-Instruct-abliterated Vision-Language Model in two ready-to-use formats for ComfyUI:

Source / Provenance

• Base model: huihui-ai/Huihui-Qwen3-VL-4B-Instruct-abliterated (apache-2.0)
• Origin model: Qwen/Qwen3-VL-4B-Instruct
• Abliteration: only the text part was processed (not the vision part), so the visual encoder behaves identically to the upstream model.

What was done

BF16 single-file (*.safetensors, 8.88 GiB)

The original upstream repo ships the weights split across two safetensors shards (model-00001-of-00002.safetensors + model-00002-of-00002.safetensors). They were merged into a single safetensors file using the original model.safetensors.index.json mapping. No weights modified.

• 713 tensors
• dtype: bfloat16
• Verified structurally identical to upstream (same key set)

FP8 scaled (*-fp8_scaled.safetensors, 5.24 GiB)

Per-tensor abs-max quantization to float8_e4m3fn for the 252 linear projections of the language model (q/k/v/o_proj + gate/up/down_proj across all layers). Embeddings, layer norms, biases and the entire visual encoder stay in BF16.

• 1217 tensors (252 × 3 + 461 BF16)
• Quantised layers: float8_e4m3fn weights + float32 per-tensor scale + uint8[64] comfy_quant marker (JSON: {"format": "float8_e4m3fn", "full_precision_matrix_mult": false})
• Per-tensor scale = max(|w|) / 448 (E4M3FN max)
• Mean round-trip relative error ≈ 2.3% (typical for FP8 LLM quantisation)
• Schema matches the ComfyUI "fp8_scaled" convention used by other models in this size class (e.g. qwen3vl_4b_fp8_scaled.safetensors)

INT8 ConvRot (*-int8_convrot.safetensors, 4.72 GiB)

Row-wise INT8 quantization of the 252 linear projections of the language model, with Hadamard rotation (group size 256) applied to the weight matrices to distribute outliers before quantization. Embeddings, layer norms, biases, the entire visual encoder and the first transformer block stay in BF16 (protected by the --qwen35 filter). Quantized directly from the BF16 source (not from FP8) using silveroxides/convert_to_quant.

• 1724 tensors (358 quantized × 3 + 650 BF16)
• Per-tensor .comfy_quant JSON marker: {"format": "int8_tensorwise", "orig_dtype": "torch.bfloat16", "convrot": true, "convrot_groupsize": 256, "per_row": true}
• weight_scale shape: per-row (N, 1), not scalar — the per-row scale is what makes ConvRot compatible with LoRA application at runtime
• Quantization command (reproducible):

  ctq -i Huihui-Qwen3-VL-4B-Instruct-abliterated.safetensors \
      -o Huihui-Qwen3-VL-4B-Instruct-abliterated-int8_convrot.safetensors \
      --int8 --convrot --convrot-group-size 256 \
      --scaling_mode row \
      --comfy_quant --save-quant-metadata --qwen35 \
      --simple --low-memory --device cuda
  

• Quantization was done on GPU (NVIDIA RTX 3090) in ~3 minutes.

Why INT8 ConvRot over FP8 on RTX 30-series

On Ampere GPUs (RTX 3090) FP8 is emulated in software because Ampere has no native FP8 tensor cores. INT8 has native INT8 tensor cores on every RTX 30+ generation. Result on RTX 3090: ~20-30% faster text-encoder forward pass than the FP8-scaled variant, with comparable quality.

Why --scaling_mode row matters

The default --scaling_mode tensor produces a single global scale per weight matrix (weight_scale shape ()). For ConvRot's per-row Hadamard rotation to remain correct on the quantization side, per-row scales are required (weight_scale shape (N, 1), one scale per output channel). Without --scaling_mode row, the file loads but LoRAs applied at runtime silently degrade to plain tensorwise INT8. Always verify after conversion:

from safetensors import safe_open
import json
with safe_open("…-int8_convrot.safetensors", framework="pt") as f:
    raw = f.get_tensor([k for k in f.keys() if k.endswith('.comfy_quant')][0]).tolist()
    print(json.loads(bytes(raw)))
# Must contain: convrot=True, per_row=True, convrot_groupsize=256

Quantisation script

The FP8 conversion was done on GPU (NVIDIA RTX 3090) in ~4 seconds. Script is available on request.

Usage

ComfyUI — Qwen3-VL Text Encoder (recommended)

1. Drop the *-fp8_scaled.safetensors into your ComfyUI models/text_encoders/ directory.
2. Use the Qwen3-VL Text Encoder node and select Huihui-Qwen3-VL-4B-Instruct-abliterated-fp8_scaled.
3. Pair with a Qwen3-VL compatible diffusion model and sampler.

ComfyUI — INT8 ConvRot (recommended on RTX 30/40/50)

1. Drop the *-int8_convrot.safetensors into your ComfyUI models/text_encoders/ directory.
2. Use the native Qwen3-VL Text Encoder node and select the INT8 file. (ComfyUI ≥ 0.27.0 supports INT8 ConvRot natively; no custom node required for this text-encoder path.)
3. ~20-30% faster forward pass than the FP8-scaled variant on RTX 30-series (native INT8 tensor cores). Pair with any Qwen3-VL compatible workflow — this encoder is not Krea-specific.

ComfyUI — Full BF16 (when more precision is required)

1. Drop the *.safetensors into models/text_encoders/.
2. Use the same node but select the BF16 file. Higher VRAM usage (~16 GB on top of the diffusion model for FP16 diffusion).

transformers (BF16 only — tokenizer/configs are not bundled here)

The upstream repo huihui-ai/Huihui-Qwen3-VL-4B-Instruct-abliterated has the matching tokenizer, processor and configs. For BF16 inference:

from transformers import AutoProcessor, Qwen3VLForConditionalGeneration
import torch

model = Qwen3VLForConditionalGeneration.from_pretrained(
    "ahmed22xa/Huihui-Qwen3-VL-4B-Instruct-abliterated-comfy",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
processor = AutoProcessor.from_pretrained("huihui-ai/Huihui-Qwen3-VL-4B-Instruct-abliterated")

The FP8 file is not loadable with transformers.from_pretrained directly — it follows ComfyUI's per-tensor-FP8 layout with comfy_quant markers.

License & disclaimer

• License: apache-2.0 (inherited from upstream Qwen/Qwen3-VL-4B-Instruct).
• Abliteration notice: This is an uncensored variant. The safety filtering has been significantly reduced, potentially generating sensitive, controversial, or inappropriate content. Use with caution. See the upstream model card for the full disclaimer.
• No warranty. Users are solely responsible for any consequences arising from use of this model.