---
license: apache-2.0
language:
- en
- zh
- multilingual
library_name: gguf
tags:
- gguf
- nex
- nex-n2
- moe
- mixture-of-experts
- qwen3_5_moe
- vision
- multimodal
- nvfp4
- NVFP4
base_model: nex-agi/Nex-N2-mini
pipeline_tag: image-text-to-text
inference: false
quantized_by: FreedomAI
---

# Nex-N2-mini-NVFP4-GGUF

GGUF quantization of [nex-agi/Nex-N2-mini](https://huggingface.co/nex-agi/Nex-N2-mini) — a 35B MoE agentic model (3B active) built on Qwen3.5-35B-A3B-Base with 256 experts, Gated DeltaNet hybrid attention, 262K context, and 27-layer vision encoder.

Quantized to **NVFP4** format for efficient inference with minimal quality loss.

## About NVFP4

NVFP4 is NVIDIA's native 4-bit floating-point format (E4M3) for Blackwell GPUs. It stores weights in FP4 with a shared per-block scale, enabling native Blackwell tensor core acceleration with no dequantization overhead during inference.

## Files

| Filename | Type | Size | Description |
|----------|------|------|-------------|
| `nex-n2-mini-nvfp4.gguf` | GGUF (NVFP4) | 18.36 GB | Quantized text model weights |
| `mmproj-nex-n2-mini-f16.gguf` | F16 mmproj | 0.84 GB | Vision encoder projector (27-layer ViT, 1152 hidden) |
| `README.md` | Markdown | - | Model card |

## Quantization Details

| Property | Value |
|----------|-------|
| Format | NVFP4 |
| Bits Per Weight | 4.55 BPW |
| File Size | 18.36 GB (text) + 0.84 GB (mmproj) |
| Tensor Count | 733 (text) + 334 (mmproj) |
| Architecture | qwen3_5_moe |

## Model Description

- **Developer**: Nex AGI
- **Base Model**: Qwen3.5-35B-A3B-Base
- **Architecture**: Mixture-of-Experts (MoE) with Gated DeltaNet + full attention
- **Parameters**: 35B total, 3B activated per token
- **Experts**: 256 routed experts (8 per token) + 1 shared
- **Context Length**: 262,144 tokens
- **Vision**: 27-layer ViT encoder (1152 hidden), image-text-to-text
- **Languages**: English, Chinese, multilingual
- **License**: Apache 2.0

## Usage

### llama.cpp (CLI)

```bash
# Text + Image
llama-cli -m nex-n2-mini-nvfp4.gguf \
  --mmproj mmproj-nex-n2-mini-f16.gguf \
  --image photo.jpg \
  -p "Describe this image in detail" \
  -n 512

# Text only
llama-cli -m nex-n2-mini-nvfp4.gguf \
  -p "Explain quantum computing in simple terms" \
  -n 512

# OpenAI-compatible server
llama-server -m nex-n2-mini-nvfp4.gguf \
  --mmproj mmproj-nex-n2-mini-f16.gguf \
  --port 8080
```

### llama-cpp-python

```python
from llama_cpp import Llama

llm = Llama.from_pretrained(
    repo_id="FreedomAISVR/Nex-N2-mini-NVFP4-GGUF",
    filename="nex-n2-mini-nvfp4.gguf",
    n_gpu_layers=-1,
)

response = llm.create_chat_completion([
    {"role": "user", "content": "What is the capital of France?"}
])
print(response["choices"][0]["message"]["content"])
```

### Direct download

```python
from huggingface_hub import hf_hub_download

for filename in ["nex-n2-mini-nvfp4.gguf", "mmproj-nex-n2-mini-f16.gguf"]:
    hf_hub_download(
        repo_id="FreedomAISVR/Nex-N2-mini-NVFP4-GGUF",
        filename=filename,
        local_dir="./models"
    )
```

## Quantization Pipeline

```
1. Download source weights
   huggingface_hub.snapshot_download("nex-agi/Nex-N2-mini")

2. Convert text model to F16 GGUF
   convert_hf_to_gguf.py --outtype f16

3. Extract vision encoder
   convert_hf_to_gguf.py --mmproj --outtype f16

4. Quantize to NVFP4
   llama-quantize  nex-n2-mini-f16.gguf nex-n2-mini-nvfp4.gguf NVFP4
```

## Hardware

| Component | Specification |
|-----------|---------------|
| GPU | NVIDIA RTX 5060 Ti (Blackwell) |
| System RAM | 64 GB |
| Storage | NVMe |

## License

Apache 2.0 — same as the original [nex-agi/Nex-N2-mini](https://huggingface.co/nex-agi/Nex-N2-mini).