Instructions to use dtometzki/Qwen3-30B-A3B-MXFP4A16 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use dtometzki/Qwen3-30B-A3B-MXFP4A16 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="dtometzki/Qwen3-30B-A3B-MXFP4A16")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("dtometzki/Qwen3-30B-A3B-MXFP4A16")
model = AutoModelForCausalLM.from_pretrained("dtometzki/Qwen3-30B-A3B-MXFP4A16", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use dtometzki/Qwen3-30B-A3B-MXFP4A16 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "dtometzki/Qwen3-30B-A3B-MXFP4A16"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "dtometzki/Qwen3-30B-A3B-MXFP4A16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/dtometzki/Qwen3-30B-A3B-MXFP4A16

SGLang

How to use dtometzki/Qwen3-30B-A3B-MXFP4A16 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "dtometzki/Qwen3-30B-A3B-MXFP4A16" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "dtometzki/Qwen3-30B-A3B-MXFP4A16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "dtometzki/Qwen3-30B-A3B-MXFP4A16" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "dtometzki/Qwen3-30B-A3B-MXFP4A16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use dtometzki/Qwen3-30B-A3B-MXFP4A16 with Docker Model Runner:
```
docker model run hf.co/dtometzki/Qwen3-30B-A3B-MXFP4A16
```

Qwen3-30B-A3B-MXFP4A16 / README.md

dtometzki

Update README.md

d5a7afb verified 8 months ago

preview code

Raw

History Blame Contribute Delete

3.44 kB

	---
	base_model: Qwen/Qwen3-30B-A3B
	model_name: Qwen3-30B-A3B-MXFP4A16
	library_name: transformers
	license: apache-2.0
	language:
	- en
	- zh
	tags:
	- quantization
	- w4a16
	- mxfp4
	- fp4
	- compressed-tensors
	- llmcompressor
	- text-generation
	- safetensors
	pipeline_tag: text-generation
	inference: false
	---

	```markdown
	base_model: Qwen/Qwen3-30B-A3B
	library_name: transformers
	tags:
	- quantization
	- mxfp4
	- 4-bit
	- compressed-tensors
	- qwen
	- text-generation
	- llmcompressor
	language:
	- en
	pipeline_tag: text-generation
	license: other
	---

	# Qwen3-30B-A3B-MXFP4A16

	## Model Description

	This is a compressed version of [Qwen/Qwen3-30B-A3B](https://huggingface.co/Qwen/Qwen3-30B-A3B).

	The model was quantized using Weight-Only Quantization to 4-bit Floating Point (FP4) using the MXFP4 scaling scheme. This format is optimized for next-generation hardware (like NVIDIA Blackwell) but also runs efficiently on current GPUs using software emulation.

	By using `MXFP4A16` (Microscaling FP4 weights with FP16 activations), this model achieves a massive reduction in size (approx. 70-75%) while maintaining high accuracy, especially for weights distributed around zero, compared to standard INT4 quantization.

	## Quantization Details

	This model was created using the `llmcompressor` library with the following configuration:

	* Scheme: `MXFP4A16` (4-bit Weights, 16-bit Activations)
	* Algorithm: Weight-Only Quantization (Data-Free)
	* Target Modules: Linear Layers
	* Ignored Modules: `lm_head` (kept in full precision for stability)
	* Group Size: 32 (Block-wise scaling)

	## Installation

	You need to install `vllm` or `llmcompressor` to use this model efficiently.

	```bash
	pip install vllm
	# or
	pip install llmcompressor

	```

	## Quickstart

	### Using vLLM (Recommended)

	This model is optimized for vLLM, which supports the `compressed-tensors` format natively.

	```python
	from vllm import LLM, SamplingParams

	model_id = "DEIN_USERNAME/Qwen3-30B-A3B-MXFP4A16"

	llm = LLM(
	model=model_id,
	trust_remote_code=True
	)

	sampling_params = SamplingParams(temperature=0.7, top_p=0.9, max_tokens=200)

	prompts = [
	"Hello, my name is",
	"Explain quantum physics in simple terms:",
	]

	outputs = llm.generate(prompts, sampling_params)

	for output in outputs:
	prompt = output.prompt
	generated_text = output.outputs[0].text
	print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

	```

	### Using Transformers & LLMCompressor

	```python
	from transformers import AutoModelForCausalLM, AutoTokenizer
	from llmcompressor.utils import dispatch_for_generation

	model_id = "DEIN_USERNAME/Qwen3-30B-A3B-MXFP4A16"

	model = AutoModelForCausalLM.from_pretrained(
	model_id,
	device_map="auto",
	trust_remote_code=True
	)
	tokenizer = AutoTokenizer.from_pretrained(model_id)

	# Optimize model for generation
	dispatch_for_generation(model)

	input_ids = tokenizer("Hello my name is", return_tensors="pt").input_ids.to(model.device)
	output = model.generate(input_ids, max_new_tokens=100)

	print(tokenizer.decode(output[0]))

	```

	## Hardware Requirements

	* VRAM: Significantly reduced compared to the BF16 original.
	* Original (30B): ~60 GB VRAM
	* Quantized (MXFP4): ~17 GB VRAM


	* Compatibility: Runs on NVIDIA GPUs (Ampere/Ada Lovelace/Hopper/Blackwell).

	---

	Created with [llmcompressor](https://www.google.com/search?q=https://github.com/neuralmagic/llm-compressor)

	```