Qwen3-VL-Embedding-8B-mlx-8bit

TLDR: 8-bit quantized Qwen3-VL embedding model for Apple Silicon — produces 4096-dimensional multimodal embeddings for text and images via MLX/Metal. This model is designed for use with MLX and mlx-vlm on Apple Silicon. It is not a standard Hugging Face transformers model — it uses MLX's native quantization format and safetensors layout.

Mixed-precision MLX quantization of Qwen's 8B multimodal embedding model. The language model projections are compressed to 8-bit while the vision tower and all norm layers remain at full BF16 precision, preserving multimodal embedding quality at a fraction of the memory footprint.

Source model: https://huggingface.co/Qwen/Qwen3-VL-Embedding-8B

---

(1) Introduction

The Objective: Deploy Qwen's 8B multimodal embedding model on Apple Silicon for high-performance, on-device semantic search over mixed text and image content.

The Problem: The BF16 model requires ~16 GB of unified memory just for weights. Uniform 4-bit quantization degrades the vision tower and collapses multimodal embedding quality.

The Solution: Mixed Quantization. A surgical approach that compresses the language model's projection matrices — MLP (gate_proj, up_proj, down_proj) and attention (q_proj, k_proj, v_proj, o_proj, lm_head, embed_tokens) — to 8-bit affine quantization with group_size=64, while preserving the entire 27-layer vision tower and all normalization layers at full BF16 precision.

The result: a 9.18 GB model that fits comfortably in unified memory alongside the inference runtime, producing 4096-dimensional embeddings for both text and images with no measurable quality loss.

Version	Size	Notes
BF16 baseline	~16 GB	Full precision
Mixed 8-bit (this model)	9.18 GB	~43% reduction, vision tower protected at BF16

---

(2) Architecture

Qwen3-VL-Embedding-8B is a vision-language model built on the Qwen3-VL architecture, adapted for embedding generation rather than text generation. It uses the same Qwen3VLForConditionalGeneration architecture as the generative variant but with a modified chat template ("Represent the user's input.") and is used to extract hidden states as embeddings.

Language Model (Text)

Parameter	Value
Hidden size	4096
Intermediate size (SwiGLU)	12288
Layers	36
Attention heads	32
KV heads (GQA)	8
Head dimension	128
Vocabulary	151,936
Max context	262,144 tokens
RoPE theta	5,000,000
Activation	SiLU (SwiGLU)

Vision Tower

Parameter	Value
Depth	27 blocks
Hidden size	1152
Intermediate size	4304
Attention heads	16
Patch size	16
Spatial merge size	2
Output hidden size	4096
Deepstack merger indexes	[8, 16, 24]
Max image resolution	1,310,720 pixels (longest edge)
Min image resolution	4,096 pixels (shortest edge)

Embedding Extraction

The model produces embeddings by extracting the final hidden state from the last token position (shape: [batch, 4096]) after passing text or multimodal input through the full architecture. The lm_head projection is replaced with an identity function at runtime so that raw hidden states are returned instead of vocabulary logits. Embeddings are L2-normalized before output.

---

(3) Quantization Strategy

The Mixed-Precision Split

Component	Precision	Rationale
Vision tower (all 27 blocks, merger, deepstack mergers, patch embed, pos embed)	BF16	Image comprehension is the foundation of multimodal embeddings. Quantizing the vision encoder collapses cross-modal alignment.
Norm layers (input_layernorm, post_attention_layernorm, q_norm, k_norm, final norm)	BF16	Stability anchors. Negligible size (~145 tensors). Quantizing norms introduces instability in the embedding space.
MLP projections (gate_proj, up_proj, down_proj × 36 layers)	8-bit	SwiGLU MLPs are the most quantization-tolerant component. These are large feature transformation matrices that absorb 8-bit with minimal penalty.
Attention projections (q_proj, k_proj, v_proj, o_proj × 36 layers)	8-bit	Attention weights benefit from 8-bit compression. Group_size=64 preserves enough precision for cross-token attention quality.
Embed tokens	8-bit	Input embeddings compress well at 8-bit.
LM head	8-bit	Not used for generation (replaced with Identity at runtime), but stored in quantized form.

Quantization Parameters

• Scheme: Affine quantization (scales + biases per group)
• Bits: 8
• Group size: 64
• Format: Native MLX safetensors with per-tensor quantization metadata in tensor headers

Weight Distribution

Category	Tensor count	Precision
Vision tower (BF16)	351	BF16
Language model projections (quantized)	254 layer groups (508 tensors incl. biases/scales)	8-bit
Language model norms (BF16)	145	BF16
Total model size	—	9.18 GB (2 shards)

---

(4) Deployment

This model is designed for use with MLX and mlx-vlm on Apple Silicon. It is not a standard Hugging Face transformers model — it uses MLX's native quantization format and safetensors layout.

Prerequisites

• Hardware: Apple Silicon. M1 with 16 GB RAM is the minimum recommended.
• Python: 3.12+
• Dependencies: mlx, mlx-vlm, fastapi, uvicorn, neo4j, pillow, torch (for tensor conversion in preprocessing)

Loading the Model (example)

import mlx.core as mx
import mlx.nn as nn
from mlx_vlm.utils import load

# Load the model (lazy loading recommended for memory efficiency)
model, processor = load("your-username/Qwen3-VL-Embedding-8B-mlx-8bit", lazy=True)

# Replace lm_head with Identity to extract raw hidden states
class Identity(nn.Module):
    def __call__(self, x):
        return x

model.language_model.lm_head = Identity()

Generating Text Embeddings (example)

import torch
import mlx.core as mx

def embed_text(text: str) -> list:
    """Generate a 4096-dimensional L2-normalized embedding for text."""
    # Tokenize
    inputs = processor.tokenizer([text], return_tensors="pt", padding=True)

    # Convert PyTorch tensors to MLX arrays
    inputs_mx = {}
    for key, value in inputs.items():
        if isinstance(value, torch.Tensor):
            inputs_mx[key] = mx.array(value.numpy())
        else:
            inputs_mx[key] = value

    # Forward pass — outputs.logits now contains raw hidden states
    outputs = model(**inputs_mx)

    # Extract last token's hidden state: [batch, seq_len, 4096] -> [batch, 4096]
    raw_embeddings = outputs.logits[:, -1, :]

    # L2 normalize
    norms = mx.linalg.norm(raw_embeddings, axis=-1, keepdims=True)
    normalized = raw_embeddings / norms

    # Convert to Python list
    return normalized[0].tolist()

embedding = embed_text("Your text here")
print(f"Embedding dimension: {len(embedding)}")  # 4096

Generating Image Embeddings (example)

from PIL import Image
import torch
import mlx.core as mx

def embed_image(image_path: str, anchor_text: str = "Extract visual features.") -> list:
    """Generate a 4096-dimensional L2-normalized embedding for an image."""
    image = Image.open(image_path).convert('RGB')

    # Build multimodal message
    messages = [
        {
            "role": "user",
            "content": [
                {"type": "image", "image": image},
                {"type": "text", "text": anchor_text}
            ]
        }
    ]

    # Apply chat template
    text_prompt = processor.apply_chat_template(
        messages, tokenize=False, add_generation_prompt=True
    )

    # Process with images
    inputs = processor(text=[text_prompt], images=[image], padding=True, return_tensors="pt")

    # Convert to MLX
    inputs_mx = {}
    for k, v in inputs.items():
        if v is not None and hasattr(v, 'numpy'):
            inputs_mx[k] = mx.array(v.numpy())
        elif v is not None:
            inputs_mx[k] = mx.array(v)

    # Forward pass
    outputs = model(**inputs_mx)

    # Extract and normalize
    raw_embeddings = outputs.logits[:, -1, :]
    norms = mx.linalg.norm(raw_embeddings, axis=-1, keepdims=True)
    normalized = raw_embeddings / norms

    return normalized[0].tolist()

embedding = embed_image("/path/to/image.jpg", anchor_text="diagram, architecture, technical")

Running the Embedding Server (example)

# Set environment variables
export MODEL_PATH="/path/to/Qwen3-VL-Embedding-8B-mlx-8bit"
export NEO4J_URI="neo4j://localhost:7687"
export NEO4J_USERNAME="username"
export NEO4J_PASSWORD="your-password"

# Start the server
cd embedding_server
python3 -m uvicorn main:app --host 0.0.0.0 --port 8000 --log-level info

---

(5) Critical Deployment Notes

The lm_head Identity Patch Is Required

This model is an embedding model, not a generation model. The lm_head layer must be replaced with an identity function at runtime to extract raw hidden states:

class Identity(nn.Module):
    def __call__(self, x):
        return x

model.language_model.lm_head = Identity()

Without this patch, the model will return vocabulary logits (151,936 dimensions) instead of the desired 4096-dimensional hidden states.

PyTorch Is Required for Preprocessing

The mlx-vlm processor uses Hugging Face's transformers library under the hood, which returns PyTorch tensors. These must be converted to MLX arrays before passing to the model:

# This conversion step is mandatory
inputs_mx = {}
for key, value in inputs.items():
    if isinstance(value, torch.Tensor):
        inputs_mx[key] = mx.array(value.numpy())
    else:
        inputs_mx[key] = value

Note on config.json

The config.json declares "bits": 8 with "mode": "affine", but MLX reads per-layer quantization metadata dynamically from the safetensors tensor headers. The actual mixed-precision structure — which layers are 8-bit and which remain BF16 — lives in the safetensors files, not the config. Tooling that trusts only config.json will see a uniform 8-bit model; this is expected behavior for native MLX mixed quantization.

---

(6) Model Files

File	Size	Description
model-00001-of-00002.safetensors	4.98 GB	Layers 0–16, vision tower, tokenizer
model-00002-of-00002.safetensors	4.20 GB	Layers 17–35, lm_head
model.safetensors.index.json	119 KB	Weight index and shard map
config.json	2 KB	Model architecture and quantization config
processor_config.json	1 KB	Processor (tokenizer + image processor) config
preprocessor_config.json	784 B	Image preprocessing parameters
tokenizer.json	10.9 MB	Tokenizer model
tokenizer_config.json	389 B	Tokenizer configuration
chat_template.jinja	5.4 KB	Chat template with embedding system message
vocab.json	2.65 MB	Vocabulary
special_tokens_map.json	613 B	Special token mappings
added_tokens.json	707 B	Added special tokens
README.md	14 KB	This file

---

(7) Use Cases

This model is designed for multimodal semantic search and vector database ingestion:

• Document search: Embed PDF pages, text content, and embedded images into a unified 4096-dimensional space for cross-modal retrieval.
• Image search: Generate keyword-anchored embeddings for images and visual crops, enabling natural language search over image collections.
• Code search: Embed source code files alongside documentation and screenshots for developer knowledge bases.
• Agentic workflows: Provide on-device, low-latency embeddings for AI agents that need to search local document stores without cloud API calls.

---

(8) License

This model inherits the Apache 2.0 license from the base Qwen/Qwen3-VL-Embedding-8B model.

---

Notes

• This is a native MLX quantization — it is not compatible with transformers, bitsandbytes, or other non-MLX inference frameworks.
• The model requires mlx-vlm for loading. Standard mlx_lm will not work due to the vision-language architecture.
• Embedding dimension is 4096 (the language model's hidden size), not the vocabulary size.
• The chat template default system message is "Represent the user's input." — this is the embedding instruction that guides the model to produce representation vectors rather than text completions.