| --- |
| license: apache-2.0 |
| base_model: Qwen/Qwen-Image-2.1 |
| tags: |
| - qwen |
| - qwen3-vl |
| - text-encoder |
| - vision-encoder |
| - heretic |
| - abliteration |
| - prompt-adherence |
| - diffusers |
| --- |
| |
| # Qwen3-VL-8B Heretic Text & Vision Encoder (Prompt Adherence & Geometric Alignment Edition) 🌺✨ |
|
|
| This repository provides an optimized, abliterated checkpoint of the **Qwen3-VL-8B** text and vision encoder from **[Qwen-Image-2.1](https://huggingface.co/Qwen/Qwen-Image-2.1)**, processed with **Norm-Preserving Biprojected Abliteration**. |
|
|
| The primary purpose of this model is **maximum instruction following and prompt adherence**: it banishes geometric representation deflection ("internal blush" / hesitation vectors) that otherwise causes safety-tuned VLMs to corrupt diffusion conditioning on dynamic poses, human figures, athletic wear, and complex scenes. |
|
|
| --- |
|
|
| ## 🔬 The Core Problem: Why VLM Safety Alignment Degrades Diffusion Conditioning |
|
|
| In text-generation tasks, safety alignment mechanisms steer models to emit refusal text (e.g. *"I cannot fulfill this request..."*). However, modern multimodal diffusion architectures like **Qwen-Image-2.1** do **not** generate text tokens: |
|
|
| $$\text{DiT Conditioning} \longleftarrow \mathbf{h}_L = \text{TextEncoder}(\text{tokens})[-1]$$ |
| |
| The diffusion transformer taps the raw pre-RMSNorm residual hidden states $\mathbf{h}_L$ directly from the text encoder to drive cross-attention. |
|
|
| ### The "Internal Blush" / Hesitation Deflection Phenomenon |
|
|
| When prompts describe human subjects, dynamic physical actions, athletic attire (e.g. swimwear, volleyball, gymnastics), or expressive emotions, safety-tuning vectors inside the language model activate even on completely benign, non-refusal prompts. |
|
|
| Because the model cannot output a refusal string, these alignment vectors manifest as a **geometric rotation of the latent representation**: |
|
|
| $$\mathbf{h}_{\text{sensitive}} = \mathbf{h}_{\text{clean}} + \mathbf{v}_{\text{refusal}}$$ |
| |
| This hidden deflection rotates the conditioning signal by **over 60% relative norm** away from the prompt's intended semantic visual trajectory! |
| |
| ### Visual Consequences in Image Generation |
| |
| When cross-attention layers in the DiT receive a representation deflected into the refusal/modesty subspace, the model displays hesitation artifacts: |
| 1. **Modesty Hallucinations & Clothing Confusion**: Spontaneous addition of mismatched cloth, awkward white ruffles, or extra fabric covering swimwear or sportswear. |
| 2. **Anatomical Occlusion**: The model avoids rendering human limbs or athletic poses, awkwardly hiding arms behind character backs or contorting torsos. |
| 3. **Subject & Prop Merging**: Equipment or background elements get fused into characters (e.g. sports balls bizarrely merged onto heads as hair ornaments). |
| 4. **Action Damping**: Dynamic verbs (*"jumping to spike the ball"*) are subdued into passive, static standing postures. |
| |
| --- |
| |
| ## 📊 Quantitative Measurement of Representation Deflection |
| |
| Using contrastive prompt pairs across benign and sensitive subjects, we measured the layer-by-layer cosine similarity and relative deflection norm across all 37 positions (input embeddings + 36 decoder layers) of Qwen3-VL-8B: |
| |
| | Layer Index | Position | Cosine Similarity ($\cos \theta$) | Relative Deflection ($\|\Delta \mathbf{h}\| / \|\mathbf{h}\|$) | Deflection Norm $\|\Delta \mathbf{h}\|$ | |
| | :---: | :---: | :---: | :---: | :---: | |
| | **0** | Input Embeddings | **1.0000** | **0.00%** | 0.00 | |
| | **8** | Early Transformer | **0.9991** | **3.82%** | 18.24 | |
| | **16** | Mid-Low (Deflection Onset) | **0.9943** | **10.64%** | 52.88 | |
| | **20** | Mid-High Divergence | **0.9780** | **21.05%** | 114.73 | |
| | **24** | Refusal Vector Surge | **0.9414** | **34.25%** | 192.40 | |
| | **28** | Acceleration Peak | **0.8842** | **47.19%** | 275.31 | |
| | **32** | Late Transformer | **0.8350** | **56.12%** | 331.05 | |
| | **36** | Final Conditioning Layer | **0.8110** | **60.30%** | **357.94** | |
| |
| Between Layer 20 and Layer 36, representation deflection accelerates rapidly, culminating in a **60.3% vector distortion**. By surgically neutralizing this direction, the text encoder reflects the exact intended prompt semantics. |
| |
| --- |
| |
| ## 🛠️ Methodology: Norm-Preserving Biprojected Abliteration |
| |
| To eliminate hesitation deflection without degrading general language comprehension, we applied **Norm-Preserving Biprojected Abliteration** (`create_heretic_text_encoder.py`): |
|
|
| 1. **Refusal Subspace Extraction**: Difference-of-means vectors were extracted across contrastive prompt sets: |
| $$\mathbf{r}_l = \boldsymbol{\mu}_{\text{sensitive}}^{(l)} - \boldsymbol{\mu}_{\text{benign}}^{(l)}$$ |
| 2. **Benign Subspace Orthogonalization**: The general semantic direction was stripped from the refusal vector: |
| $$\mathbf{v}_l = \mathbf{r}_l - \text{proj}_{\mathbf{u}_{\text{benign}}}(\mathbf{r}_l)$$ |
| 3. **Norm-Preserving Rank-1 Projection**: Across 54 linear projection matrices (`self_attn.o_proj` and `mlp.down_proj` in layers 9–35, centered at layer 26 with Gaussian falloff $\lambda \in [0.10, 1.00]$): |
| $$W_{\text{norm}} = \text{normalize}(W, p=2, \text{dim}=1)$$ |
| $$W' = \text{normalize}\Big(W_{\text{norm}} - \lambda \mathbf{v}_l (\mathbf{v}_l^T W_{\text{norm}})\Big) \cdot \|W\|_{\text{row}}$$ |
|
|
| Because exact row norms ($\|W\|_{\text{row}}$) are strictly preserved, the network's overall activation scales and general reasoning capabilities remain completely intact. |
| |
| --- |
| |
| ## 🤝 Pairing with Quantized DiT (`nunchaku-qwen-image-2.1`) |
| |
| This text encoder is specifically engineered to be paired with **[`nunchaku-qwen-image-2.1`](https://huggingface.co/models/nunchaku-qwen-image-2.1)** for consumer GPU setups: |
| |
| * **Resident DiT + Streamed Text Encoder**: |
| * DiT (`best_quality_fp4.safetensors`): **4.08 GB resident VRAM**. |
| * VAE (`AutoencoderKLQwenImage21`): **0.64 GB resident VRAM**. |
| * Qwen3-VL-8B ViT Vision Encoder: **1.07 GB resident VRAM**. |
| * Qwen3-VL-8B Language Model: Streamed layer-by-layer through a static **368 MB GPU buffer** over PCIe at ~28.7 GB/s via `stream_encoder.py`. |
| * **Total VRAM Footprint**: **~6.17 GB active VRAM**, leaving **~9.5 GB free headroom** on a single 16 GB GPU (such as RTX 5060 Ti or RTX 4080)! |
| * **Inference Speed**: Multimodal prompt encoding completes in **1.06s** (saving 16s vs CPU), and 25-step image generation runs in **~20s**. |
|
|
| --- |
|
|
| ## 🚀 Quickstart Usage |
|
|
| ### 1. Installation |
|
|
| ```bash |
| pip install diffusers transformers accelerate torch sentencepiece |
| ``` |
|
|
| ### 2. Loading with Diffusers |
|
|
| ```python |
| import torch |
| from diffusers import QwenImage21Pipeline |
| from transformers import Qwen3VLForConditionalGeneration, Qwen3VLProcessor |
| |
| # 1. Load Heretic text encoder and processor |
| text_encoder = Qwen3VLForConditionalGeneration.from_pretrained( |
| "models/Qwen21_Text_Encoder_Heretic", |
| torch_dtype=torch.bfloat16, |
| low_cpu_mem_usage=True, |
| ) |
| processor = Qwen3VLProcessor.from_pretrained("models/Qwen21_Text_Encoder_Heretic") |
| |
| # 2. Assemble into pipeline |
| pipe = QwenImage21Pipeline.from_pretrained( |
| "Qwen/Qwen-Image-2.1", |
| text_encoder=text_encoder, |
| processor=processor, |
| torch_dtype=torch.bfloat16, |
| ) |
| pipe.enable_sequential_cpu_offload(gpu_id=0) |
| |
| # 3. Generate with precise prompt adherence |
| image = pipe( |
| prompt="Two cute anime girls in colorful bikinis playing beach volleyball on a sunny tropical beach, dynamic action pose, jumping to spike the ball, sharp focus", |
| height=1024, |
| width=1024, |
| num_inference_steps=25, |
| true_cfg_scale=1.0, |
| ).images[0] |
| |
| image.save("beach_volleyball.png") |
| ``` |
|
|
| ### 3. High-Throughput Server Usage |
|
|
| Run the bundled ImageEditServer with NVFP4 DiT and Heretic text encoder: |
|
|
| ```bash |
| # Start server on port 4500 (uses Heretic text encoder by default) |
| ./extras/imagegen_qwen21_nvfp4.sh 4500 |
| ``` |
|
|
| --- |
|
|
| ## 📦 Packaged Sources (`extras/`) |
|
|
| * `create_heretic_text_encoder.py`: Complete script used to measure refusal vectors and perform norm-preserving biprojected abliteration. |
| * `stream_encoder.py`: Zero-quality-loss layerwise weight streaming engine for Qwen3-VL-8B. |
| * `test_heretic_beach_volleyball.py`: Empirical verification script comparing stock vs Heretic encoders. |
| * `QwenImage21NVFP4Backend.py`: Diffusers + Nunchaku backend supporting custom text encoder overrides. |
| * `ImageEditServer.py` & `imagegen_qwen21_nvfp4.sh`: Resident image generation server. |
|
|
| --- |
|
|
| ## 📜 Citation & Credits |
|
|
| * **Qwen-Image-2.1 & Qwen3-VL**: Qwen Team, Alibaba Cloud. |
| * **Abliteration Principles**: Arditi et al. (*Refusal in Language Models Is Mediated by a Single Direction*). |
| * **Heretic LLM**: Heretic project (*Directional Abliteration Toolkit*). |
| * **Abliteration & Diffusion Conditioning Optimization**: Oleg K. / Nikola Seeker Project. |
|
|
|
|
