Instructions to use YuYu1015/Huihui-Gemma-4-E4B-it-abliterated-NVFP4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use YuYu1015/Huihui-Gemma-4-E4B-it-abliterated-NVFP4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="YuYu1015/Huihui-Gemma-4-E4B-it-abliterated-NVFP4")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("YuYu1015/Huihui-Gemma-4-E4B-it-abliterated-NVFP4")
model = AutoModelForMultimodalLM.from_pretrained("YuYu1015/Huihui-Gemma-4-E4B-it-abliterated-NVFP4", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use YuYu1015/Huihui-Gemma-4-E4B-it-abliterated-NVFP4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "YuYu1015/Huihui-Gemma-4-E4B-it-abliterated-NVFP4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "YuYu1015/Huihui-Gemma-4-E4B-it-abliterated-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/YuYu1015/Huihui-Gemma-4-E4B-it-abliterated-NVFP4

SGLang

How to use YuYu1015/Huihui-Gemma-4-E4B-it-abliterated-NVFP4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "YuYu1015/Huihui-Gemma-4-E4B-it-abliterated-NVFP4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "YuYu1015/Huihui-Gemma-4-E4B-it-abliterated-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "YuYu1015/Huihui-Gemma-4-E4B-it-abliterated-NVFP4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "YuYu1015/Huihui-Gemma-4-E4B-it-abliterated-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use YuYu1015/Huihui-Gemma-4-E4B-it-abliterated-NVFP4 with Docker Model Runner:
```
docker model run hf.co/YuYu1015/Huihui-Gemma-4-E4B-it-abliterated-NVFP4
```

Huihui-gemma-4-E4B-it-abliterated-NVFP4

English | 繁體中文

English

NVFP4 quantization of huihui-ai/Huihui-gemma-4-E4B-it-abliterated, quantized using NVIDIA ModelOpt with NVFP4_MLP_ONLY strategy (only MLP layers quantized, attention preserved in higher precision).

Model Details

Item	Value
Architecture	Dense, Per-Layer Embeddings (PLE), ~4.1B effective parameters
Base model	google/gemma-4-E4B-it
Fine-tuned by	huihui-ai (abliteration)
Quantized by	YuYu1015
Model size	~11 GB (NVFP4)
Context length	Up to 128,000 tokens
Multimodal	Vision + Audio supported

Quantization Details

Item	Value
Method	NVIDIA ModelOpt v0.42.0
Scheme	NVFP4 (E2M1 + FP8 per-group scaling, group size 16)
Strategy	`NVFP4_MLP_ONLY` — only MLP/FFN layers quantized, all attention layers preserved
Calibration dataset	abisee/cnn_dailymail
Calibration samples	512
Hardware	NVIDIA DGX Spark (GB10, 128GB unified memory)

Layers Preserved in Higher Precision

Layer	Reason
`self_attn.*` (all layers)	Attention layers preserved for accuracy (MLP_ONLY strategy)
`lm_head`	Output head
`vision_tower.*`	Vision encoder
`audio_tower.*`	Audio encoder
`multi_modal_projector.*`	Multimodal projection
`embed_tokens`	Input embeddings

Serving with vLLM

vllm serve /path/to/model \
    --quantization modelopt \
    --served-model-name gemma-4-e4b \
    --trust-remote-code \
    --gpu-memory-utilization 0.90 \
    --max-model-len 32768 \
    --enable-prefix-caching \
    --enable-chunked-prefill \
    --language-model-only

DGX Spark (SM121) Compatibility Notes

NVFP4 on SM121 falls back to W4A16 (native W4A4 path not available, missing cvt.e2m1x2 instruction)
Use --quantization modelopt (not compressed-tensors)
--language-model-only skips vision/audio encoder profiling for text-only inference
Clear page cache before starting on UMA: sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'

Safety Warning

This model has safety filtering removed (abliterated) and may generate inappropriate content. Users are solely responsible for all consequences arising from its use.

Credits

Original Model: google/gemma-4-E4B-it by Google DeepMind
Abliteration: huihui-ai
NVFP4 Quantization: YuYu1015 on NVIDIA DGX Spark (GB10)
Quantization Tool: NVIDIA ModelOpt

繁體中文

huihui-ai/Huihui-gemma-4-E4B-it-abliterated 的 NVFP4 量化版本，使用 NVIDIA ModelOpt 的 NVFP4_MLP_ONLY 策略量化（僅量化 MLP 層，attention 保留高精度）。

模型資訊

項目	數值
架構	Dense，Per-Layer Embeddings (PLE)，約 4.1B 有效參數
基礎模型	google/gemma-4-E4B-it
微調者	huihui-ai（abliteration）
量化者	YuYu1015
模型大小	~11 GB（NVFP4）
Context 長度	最高 128,000 tokens
多模態	支援視覺 + 音訊

量化詳情

項目	數值
方法	NVIDIA ModelOpt v0.42.0
方案	NVFP4（E2M1 + FP8 逐群縮放，群組大小 16）
策略	`NVFP4_MLP_ONLY` — 僅量化 MLP/FFN 層，所有 attention 層保留高精度
校準資料集	abisee/cnn_dailymail
校準樣本數	512
量化硬體	NVIDIA DGX Spark（GB10, 128GB 統一記憶體）

保留高精度的層

層	原因
`self_attn.*`（所有層）	Attention 層保留以確保精度（MLP_ONLY 策略）
`lm_head`	輸出頭
`vision_tower.*`	視覺編碼器
`audio_tower.*`	音訊編碼器
`multi_modal_projector.*`	多模態投影層
`embed_tokens`	輸入嵌入