---
license: apache-2.0
base_model: numind/NuExtract3
tags:
- quantization
- auto-round
- gptq
- llm-compressor
- compressed-tensors
- 4bit
- vlm
- document-understanding
- structured-extraction
- ocr
pipeline_tag: image-text-to-text
---

# NuExtract3 (W4A16 Quantized via AutoRound)

W4A16 (4-bit integer weights, 16-bit activations) quantized builds of [numind/NuExtract3](https://huggingface.co/numind/NuExtract3) created using Intel's [AutoRound](https://github.com/intel/auto-round) algorithm.

---

## ⚡ Quantization Details

* **Algorithm:** [Intel AutoRound (v0.14.2)](https://github.com/intel/auto-round)
* **Precision / Scheme:** W4A16 (Group Size: 32, Symmetric: `True`)
* **Calibration:** 512 samples, sequence length 4096, 1000 tuning iterations
* **Vision Tower (`quant_nontext_module`):** Kept in **BF16** to preserve visual document parsing and OCR accuracy
* **Special Modules:** Multi-Token Prediction (`mtp`, `mtp.fc`) kept in native **bfloat16**

---

## 📦 Available Repositories

* **AutoRound:** [`Vishva007/NuExtract3-W4A16-AutoRound`](https://huggingface.co/Vishva007/NuExtract3-W4A16-AutoRound)
* **AutoGPTQ:** [`Vishva007/NuExtract3-W4A16-AutoRound-GPTQ`](https://huggingface.co/Vishva007/NuExtract3-W4A16-AutoRound-GPTQ)
* **LLM-Compressor (vLLM native):** [`Vishva007/NuExtract3-W4A16-AutoRound-LLM-Compressor`](https://huggingface.co/Vishva007/NuExtract3-W4A16-AutoRound-LLM-Compressor)

---

## 🚀 vLLM Serving & Inference

### 1. Launch Server

```bash
vllm serve Vishva007/NuExtract3-W4A16-AutoRound \
  --trust-remote-code \
  --limit-mm-per-prompt '{"image": 10, "video": 0}' \
  --chat-template-content-format openai \
  --generation-config vllm \
  --max-model-len 16384 \
  --dtype bfloat16 \
  --gpu-memory-utilization 0.90

```

### 2. Structured Extraction & OCR (Python Client)

```python
import base64
import json
from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")

def encode_image(image_path):
    with open(image_path, "rb") as f:
        return base64.b64encode(f.read()).decode("utf-8")

data_url = f"data:image/png;base64,{encode_image('document.png')}"

# Structured JSON Extraction
template = {
    "invoice_number": "verbatim-string",
    "date": "date-time",
    "total": "number",
    "line_items": [{"description": "verbatim-string", "amount": "number"}]
}

response = client.chat.completions.create(
    model="Vishva007/NuExtract3-W4A16-AutoRound",
    temperature=0.2,
    messages=[
        {"role": "user", "content": [{"type": "image_url", "image_url": {"url": data_url}}]}
    ],
    extra_body={
        "chat_template_kwargs": {
            "enable_thinking": False
        }
    }
)

print(response.choices[0].message.content)

```

> **For Markdown OCR:** Replace `template` in `chat_template_kwargs` with `"mode": "markdown"`.

---

## 📊 VRAM Benefits

* **Base Model (BF16):** ~9–11 GB VRAM
* **Quantized (W4A16 G32):** ~2.8–3.5 GB VRAM (runs comfortably on 4GB/6GB consumer GPUs)

---


### 🚀 Deploy on RunPod

One-click launch environments pre-configured with PyTorch, CUDA, and dependencies for fine-tuning or quantization.

> 🎁 **Need GPU compute?** Sign up via [RunPod](https://runpod.io?ref=iabrlp7z) and get **$5–$500 in free credits** when you add your first $10.

#### PyTorch 2.14

| Template | CUDA Version | Docker Image | Template ID | Deploy |
| :--- | :--- | :--- | :--- | :--- |
| **PyTorch 2.14 (CUDA 12.6)** | 12.6 | `vishva123/cuda-12.6-pytorch-2.14-runpod` | `d7lxsa4w9m` | [![Deploy to RunPod](https://img.shields.io/badge/RunPod-Deploy%20Pod-purple?logo=runpod&logoColor=white)](https://runpod.io/console/deploy?template=d7lxsa4w9m&ref=iabrlp7z) |
| **PyTorch 2.14 (CUDA 13.0)** | 13.0 | `vishva123/cuda-13.0-pytorch-2.14-runpod` | `yk0y6j6rpg` | [![Deploy to RunPod](https://img.shields.io/badge/RunPod-Deploy%20Pod-purple?logo=runpod&logoColor=white)](https://runpod.io/console/deploy?template=yk0y6j6rpg&ref=iabrlp7z) |
| **PyTorch 2.14 (CUDA 13.2)** | 13.2 | `vishva123/cuda-13.2-pytorch-2.14-runpod` | `gsp4gwx0nw` | [![Deploy to RunPod](https://img.shields.io/badge/RunPod-Deploy%20Pod-purple?logo=runpod&logoColor=white)](https://runpod.io/console/deploy?template=gsp4gwx0nw&ref=iabrlp7z) |

#### PyTorch 2.13

| Template | CUDA Version | Docker Image | Template ID | Deploy |
| :--- | :--- | :--- | :--- | :--- |
| **PyTorch 2.13 (CUDA 12.6)** | 12.6 | `vishva123/cuda-12.6-pytorch-2.13-runpod` | `gmlupxnxfk` | [![Deploy to RunPod](https://img.shields.io/badge/RunPod-Deploy%20Pod-purple?logo=runpod&logoColor=white)](https://runpod.io/console/deploy?template=gmlupxnxfk&ref=iabrlp7z) |
| **PyTorch 2.13 (CUDA 13.0)** | 13.0 | `vishva123/cuda-13.0-pytorch-2.13-runpod` | `y3j8xvk4f4` | [![Deploy to RunPod](https://img.shields.io/badge/RunPod-Deploy%20Pod-purple?logo=runpod&logoColor=white)](https://runpod.io/console/deploy?template=y3j8xvk4f4&ref=iabrlp7z) |
| **PyTorch 2.13 (CUDA 13.2)** | 13.2 | `vishva123/cuda-13.2-pytorch-2.13-runpod` | `vigpissn5w` | [![Deploy to RunPod](https://img.shields.io/badge/RunPod-Deploy%20Pod-purple?logo=runpod&logoColor=white)](https://runpod.io/console/deploy?template=vigpissn5w&ref=iabrlp7z) |

#### PyTorch 2.12

| Template | CUDA Version | Docker Image | Template ID | Deploy |
| :--- | :--- | :--- | :--- | :--- |
| **PyTorch 2.12 (CUDA 12.6)** | 12.6 | `vishva123/cuda-12.6-pytorch-2.12-runpod` | `ctmz86zmf0` | [![Deploy to RunPod](https://img.shields.io/badge/RunPod-Deploy%20Pod-purple?logo=runpod&logoColor=white)](https://runpod.io/console/deploy?template=ctmz86zmf0&ref=iabrlp7z) |
| **PyTorch 2.12 (CUDA 13.0)** | 13.0 | `vishva123/cuda-13.0-pytorch-2.12-runpod` | `qjko5yiwzi` | [![Deploy to RunPod](https://img.shields.io/badge/RunPod-Deploy%20Pod-purple?logo=runpod&logoColor=white)](https://runpod.io/console/deploy?template=qjko5yiwzi&ref=iabrlp7z) |
| **PyTorch 2.12 (CUDA 13.2)** | 13.2 | `vishva123/cuda-13.2-pytorch-2.12-runpod` | `ifg6xmye0f` | [![Deploy to RunPod](https://img.shields.io/badge/RunPod-Deploy%20Pod-purple?logo=runpod&logoColor=white)](https://runpod.io/console/deploy?template=ifg6xmye0f&ref=iabrlp7z) |

---

## 📚 Acknowledgments

* **Base Model:** [NuMind](https://huggingface.co/numind/NuExtract3)
* **Quantization Engine:** [Intel AutoRound](https://github.com/intel/auto-round)