Instructions to use YuYu1015/Huihui-Gemma-4-E4B-it-abliterated-NVFP4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use YuYu1015/Huihui-Gemma-4-E4B-it-abliterated-NVFP4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="YuYu1015/Huihui-Gemma-4-E4B-it-abliterated-NVFP4")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("YuYu1015/Huihui-Gemma-4-E4B-it-abliterated-NVFP4")
model = AutoModelForMultimodalLM.from_pretrained("YuYu1015/Huihui-Gemma-4-E4B-it-abliterated-NVFP4", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use YuYu1015/Huihui-Gemma-4-E4B-it-abliterated-NVFP4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "YuYu1015/Huihui-Gemma-4-E4B-it-abliterated-NVFP4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "YuYu1015/Huihui-Gemma-4-E4B-it-abliterated-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/YuYu1015/Huihui-Gemma-4-E4B-it-abliterated-NVFP4

SGLang

How to use YuYu1015/Huihui-Gemma-4-E4B-it-abliterated-NVFP4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "YuYu1015/Huihui-Gemma-4-E4B-it-abliterated-NVFP4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "YuYu1015/Huihui-Gemma-4-E4B-it-abliterated-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "YuYu1015/Huihui-Gemma-4-E4B-it-abliterated-NVFP4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "YuYu1015/Huihui-Gemma-4-E4B-it-abliterated-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use YuYu1015/Huihui-Gemma-4-E4B-it-abliterated-NVFP4 with Docker Model Runner:
```
docker model run hf.co/YuYu1015/Huihui-Gemma-4-E4B-it-abliterated-NVFP4
```

Huihui-Gemma-4-E4B-it-abliterated-NVFP4 / README.md

YuYu1015

Update README.md

a94068f verified 5 months ago

preview code

Raw

History Blame Contribute Delete

5.77 kB

	---
	license: gemma
	base_model:
	- huihui-ai/Huihui-gemma-4-E4B-it-abliterated
	base_model_relation: quantized
	library_name: transformers
	pipeline_tag: text-generation
	tags:
	- safetensors
	- gemma4
	- nvfp4
	- 4-bit
	- quantized
	- abliterated
	- dgx-spark
	- vllm
	- modelopt
	language:
	- en
	- zh
	---

	# Huihui-gemma-4-E4B-it-abliterated-NVFP4

	[English](#english) \| [繁體中文](#繁體中文)

	---

	## English

	NVFP4 quantization of [huihui-ai/Huihui-gemma-4-E4B-it-abliterated](https://huggingface.co/huihui-ai/Huihui-gemma-4-E4B-it-abliterated), quantized using NVIDIA ModelOpt with `NVFP4_MLP_ONLY` strategy (only MLP layers quantized, attention preserved in higher precision).

	### Model Details

	\| Item \| Value \|
	\|---\|---\|
	\| Architecture \| Dense, Per-Layer Embeddings (PLE), ~4.1B effective parameters \|
	\| Base model \| [google/gemma-4-E4B-it](https://huggingface.co/google/gemma-4-E4B-it) \|
	\| Fine-tuned by \| [huihui-ai](https://huggingface.co/huihui-ai) (abliteration) \|
	\| Quantized by \| [YuYu1015](https://huggingface.co/YuYu1015) \|
	\| Model size \| ~11 GB (NVFP4) \|
	\| Context length \| Up to 128,000 tokens \|
	\| Multimodal \| Vision + Audio supported \|

	### Quantization Details

	\| Item \| Value \|
	\|---\|---\|
	\| Method \| [NVIDIA ModelOpt](https://github.com/NVIDIA/Model-Optimizer) v0.42.0 \|
	\| Scheme \| NVFP4 (E2M1 + FP8 per-group scaling, group size 16) \|
	\| Strategy \| `NVFP4_MLP_ONLY` — only MLP/FFN layers quantized, all attention layers preserved \|
	\| Calibration dataset \| [abisee/cnn_dailymail](https://huggingface.co/datasets/abisee/cnn_dailymail) \|
	\| Calibration samples \| 512 \|
	\| Hardware \| NVIDIA DGX Spark (GB10, 128GB unified memory) \|

	### Layers Preserved in Higher Precision

	\| Layer \| Reason \|
	\|---\|---\|
	\| `self_attn.*` (all layers) \| Attention layers preserved for accuracy (MLP_ONLY strategy) \|
	\| `lm_head` \| Output head \|
	\| `vision_tower.*` \| Vision encoder \|
	\| `audio_tower.*` \| Audio encoder \|
	\| `multi_modal_projector.*` \| Multimodal projection \|
	\| `embed_tokens` \| Input embeddings \|

	### Serving with vLLM

	```bash
	vllm serve /path/to/model \
	--quantization modelopt \
	--served-model-name gemma-4-e4b \
	--trust-remote-code \
	--gpu-memory-utilization 0.90 \
	--max-model-len 32768 \
	--enable-prefix-caching \
	--enable-chunked-prefill \
	--language-model-only
	```

	### DGX Spark (SM121) Compatibility Notes

	- NVFP4 on SM121 falls back to W4A16 (native W4A4 path not available, missing `cvt.e2m1x2` instruction)
	- Use `--quantization modelopt` (not `compressed-tensors`)
	- `--language-model-only` skips vision/audio encoder profiling for text-only inference
	- Clear page cache before starting on UMA: `sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'`

	### Safety Warning

	This model has safety filtering removed (abliterated) and may generate inappropriate content. Users are solely responsible for all consequences arising from its use.

	### Credits

	- Original Model: [google/gemma-4-E4B-it](https://huggingface.co/google/gemma-4-E4B-it) by Google DeepMind
	- Abliteration: [huihui-ai](https://huggingface.co/huihui-ai)
	- NVFP4 Quantization: [YuYu1015](https://huggingface.co/YuYu1015) on NVIDIA DGX Spark (GB10)
	- Quantization Tool: [NVIDIA ModelOpt](https://github.com/NVIDIA/Model-Optimizer)

	---

	## 繁體中文

	[huihui-ai/Huihui-gemma-4-E4B-it-abliterated](https://huggingface.co/huihui-ai/Huihui-gemma-4-E4B-it-abliterated) 的 NVFP4 量化版本，使用 NVIDIA ModelOpt 的 `NVFP4_MLP_ONLY` 策略量化（僅量化 MLP 層，attention 保留高精度）。

	### 模型資訊

	\| 項目 \| 數值 \|
	\|---\|---\|
	\| 架構 \| Dense，Per-Layer Embeddings (PLE)，約 4.1B 有效參數 \|
	\| 基礎模型 \| [google/gemma-4-E4B-it](https://huggingface.co/google/gemma-4-E4B-it) \|
	\| 微調者 \| [huihui-ai](https://huggingface.co/huihui-ai)（abliteration） \|
	\| 量化者 \| [YuYu1015](https://huggingface.co/YuYu1015) \|
	\| 模型大小 \| ~11 GB（NVFP4） \|
	\| Context 長度 \| 最高 128,000 tokens \|
	\| 多模態 \| 支援視覺 + 音訊 \|

	### 量化詳情

	\| 項目 \| 數值 \|
	\|---\|---\|
	\| 方法 \| [NVIDIA ModelOpt](https://github.com/NVIDIA/Model-Optimizer) v0.42.0 \|
	\| 方案 \| NVFP4（E2M1 + FP8 逐群縮放，群組大小 16） \|
	\| 策略 \| `NVFP4_MLP_ONLY` — 僅量化 MLP/FFN 層，所有 attention 層保留高精度 \|
	\| 校準資料集 \| [abisee/cnn_dailymail](https://huggingface.co/datasets/abisee/cnn_dailymail) \|
	\| 校準樣本數 \| 512 \|
	\| 量化硬體 \| NVIDIA DGX Spark（GB10, 128GB 統一記憶體） \|

	### 保留高精度的層

	\| 層 \| 原因 \|
	\|---\|---\|
	\| `self_attn.*`（所有層） \| Attention 層保留以確保精度（MLP_ONLY 策略） \|
	\| `lm_head` \| 輸出頭 \|
	\| `vision_tower.*` \| 視覺編碼器 \|
	\| `audio_tower.*` \| 音訊編碼器 \|
	\| `multi_modal_projector.*` \| 多模態投影層 \|
	\| `embed_tokens` \| 輸入嵌入 \|

	### DGX Spark (SM121) 相容性說明

	- NVFP4 在 SM121 上會退回 W4A16（原生 W4A4 路徑不可用，缺少 `cvt.e2m1x2` 指令）
	- 使用 `--quantization modelopt`（非 `compressed-tensors`）
	- `--language-model-only` 跳過視覺/音訊編碼器 profiling，加速純文字推理
	- UMA 架構啟動前請先清除 page cache：`sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'`

	### 安全警告

	此模型已移除安全過濾機制（abliterated），可能產生不當內容。使用者須自行承擔所有風險與法律責任。

	### 致謝

	- 原始模型：[google/gemma-4-E4B-it](https://huggingface.co/google/gemma-4-E4B-it)，Google DeepMind
	- 去審查：[huihui-ai](https://huggingface.co/huihui-ai)
	- NVFP4 量化：[YuYu1015](https://huggingface.co/YuYu1015)，於 NVIDIA DGX Spark (GB10) 上完成
	- 量化工具：[NVIDIA ModelOpt](https://github.com/NVIDIA/Model-Optimizer)

	---
	license: gemma
	base_model:
	- huihui-ai/Huihui-gemma-4-E4B-it-abliterated
	base_model_relation: quantized
	library_name: transformers
	pipeline_tag: text-generation
	tags:
	- safetensors
	- gemma4
	- nvfp4
	- 4-bit
	- quantized
	- abliterated
	- dgx-spark
	- vllm
	- modelopt
	language:
	- en
	- zh
	---

	# Huihui-gemma-4-E4B-it-abliterated-NVFP4

	[English](#english) \| [繁體中文](#繁體中文)

	---

	## English

	NVFP4 quantization of [huihui-ai/Huihui-gemma-4-E4B-it-abliterated](https://huggingface.co/huihui-ai/Huihui-gemma-4-E4B-it-abliterated), quantized using NVIDIA ModelOpt with `NVFP4_MLP_ONLY` strategy (only MLP layers quantized, attention preserved in higher precision).

	### Model Details

	\| Item \| Value \|
	\|---\|---\|
	\| Architecture \| Dense, Per-Layer Embeddings (PLE), ~4.1B effective parameters \|
	\| Base model \| [google/gemma-4-E4B-it](https://huggingface.co/google/gemma-4-E4B-it) \|
	\| Fine-tuned by \| [huihui-ai](https://huggingface.co/huihui-ai) (abliteration) \|
	\| Quantized by \| [YuYu1015](https://huggingface.co/YuYu1015) \|
	\| Model size \| ~11 GB (NVFP4) \|
	\| Context length \| Up to 128,000 tokens \|
	\| Multimodal \| Vision + Audio supported \|

	### Quantization Details

	\| Item \| Value \|
	\|---\|---\|
	\| Method \| [NVIDIA ModelOpt](https://github.com/NVIDIA/Model-Optimizer) v0.42.0 \|
	\| Scheme \| NVFP4 (E2M1 + FP8 per-group scaling, group size 16) \|
	\| Strategy \| `NVFP4_MLP_ONLY` — only MLP/FFN layers quantized, all attention layers preserved \|
	\| Calibration dataset \| [abisee/cnn_dailymail](https://huggingface.co/datasets/abisee/cnn_dailymail) \|
	\| Calibration samples \| 512 \|
	\| Hardware \| NVIDIA DGX Spark (GB10, 128GB unified memory) \|

	### Layers Preserved in Higher Precision

	\| Layer \| Reason \|
	\|---\|---\|
	\| `self_attn.*` (all layers) \| Attention layers preserved for accuracy (MLP_ONLY strategy) \|
	\| `lm_head` \| Output head \|
	\| `vision_tower.*` \| Vision encoder \|
	\| `audio_tower.*` \| Audio encoder \|
	\| `multi_modal_projector.*` \| Multimodal projection \|
	\| `embed_tokens` \| Input embeddings \|

	### Serving with vLLM

	```bash
	vllm serve /path/to/model \
	--quantization modelopt \
	--served-model-name gemma-4-e4b \
	--trust-remote-code \
	--gpu-memory-utilization 0.90 \
	--max-model-len 32768 \
	--enable-prefix-caching \
	--enable-chunked-prefill \
	--language-model-only
	```

	### DGX Spark (SM121) Compatibility Notes

	- NVFP4 on SM121 falls back to W4A16 (native W4A4 path not available, missing `cvt.e2m1x2` instruction)
	- Use `--quantization modelopt` (not `compressed-tensors`)
	- `--language-model-only` skips vision/audio encoder profiling for text-only inference
	- Clear page cache before starting on UMA: `sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'`

	### Safety Warning

	This model has safety filtering removed (abliterated) and may generate inappropriate content. Users are solely responsible for all consequences arising from its use.

	### Credits

	- Original Model: [google/gemma-4-E4B-it](https://huggingface.co/google/gemma-4-E4B-it) by Google DeepMind
	- Abliteration: [huihui-ai](https://huggingface.co/huihui-ai)
	- NVFP4 Quantization: [YuYu1015](https://huggingface.co/YuYu1015) on NVIDIA DGX Spark (GB10)
	- Quantization Tool: [NVIDIA ModelOpt](https://github.com/NVIDIA/Model-Optimizer)

	---

	## 繁體中文

	[huihui-ai/Huihui-gemma-4-E4B-it-abliterated](https://huggingface.co/huihui-ai/Huihui-gemma-4-E4B-it-abliterated) 的 NVFP4 量化版本，使用 NVIDIA ModelOpt 的 `NVFP4_MLP_ONLY` 策略量化（僅量化 MLP 層，attention 保留高精度）。

	### 模型資訊

	\| 項目 \| 數值 \|
	\|---\|---\|
	\| 架構 \| Dense，Per-Layer Embeddings (PLE)，約 4.1B 有效參數 \|
	\| 基礎模型 \| [google/gemma-4-E4B-it](https://huggingface.co/google/gemma-4-E4B-it) \|
	\| 微調者 \| [huihui-ai](https://huggingface.co/huihui-ai)（abliteration） \|
	\| 量化者 \| [YuYu1015](https://huggingface.co/YuYu1015) \|
	\| 模型大小 \| ~11 GB（NVFP4） \|
	\| Context 長度 \| 最高 128,000 tokens \|
	\| 多模態 \| 支援視覺 + 音訊 \|

	### 量化詳情

	\| 項目 \| 數值 \|
	\|---\|---\|
	\| 方法 \| [NVIDIA ModelOpt](https://github.com/NVIDIA/Model-Optimizer) v0.42.0 \|
	\| 方案 \| NVFP4（E2M1 + FP8 逐群縮放，群組大小 16） \|
	\| 策略 \| `NVFP4_MLP_ONLY` — 僅量化 MLP/FFN 層，所有 attention 層保留高精度 \|
	\| 校準資料集 \| [abisee/cnn_dailymail](https://huggingface.co/datasets/abisee/cnn_dailymail) \|
	\| 校準樣本數 \| 512 \|
	\| 量化硬體 \| NVIDIA DGX Spark（GB10, 128GB 統一記憶體） \|

	### 保留高精度的層

	\| 層 \| 原因 \|
	\|---\|---\|
	\| `self_attn.*`（所有層） \| Attention 層保留以確保精度（MLP_ONLY 策略） \|
	\| `lm_head` \| 輸出頭 \|
	\| `vision_tower.*` \| 視覺編碼器 \|
	\| `audio_tower.*` \| 音訊編碼器 \|
	\| `multi_modal_projector.*` \| 多模態投影層 \|
	\| `embed_tokens` \| 輸入嵌入 \|

	### DGX Spark (SM121) 相容性說明

	- NVFP4 在 SM121 上會退回 W4A16（原生 W4A4 路徑不可用，缺少 `cvt.e2m1x2` 指令）
	- 使用 `--quantization modelopt`（非 `compressed-tensors`）
	- `--language-model-only` 跳過視覺/音訊編碼器 profiling，加速純文字推理
	- UMA 架構啟動前請先清除 page cache：`sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'`

	### 安全警告

	此模型已移除安全過濾機制（abliterated），可能產生不當內容。使用者須自行承擔所有風險與法律責任。

	### 致謝

	- 原始模型：[google/gemma-4-E4B-it](https://huggingface.co/google/gemma-4-E4B-it)，Google DeepMind
	- 去審查：[huihui-ai](https://huggingface.co/huihui-ai)
	- NVFP4 量化：[YuYu1015](https://huggingface.co/YuYu1015)，於 NVIDIA DGX Spark (GB10) 上完成
	- 量化工具：[NVIDIA ModelOpt](https://github.com/NVIDIA/Model-Optimizer)