Instructions to use whaoyang/gemma-3-4b-novision-quant-rk3588-1.2.1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use whaoyang/gemma-3-4b-novision-quant-rk3588-1.2.1 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="whaoyang/gemma-3-4b-novision-quant-rk3588-1.2.1")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("whaoyang/gemma-3-4b-novision-quant-rk3588-1.2.1")
model = AutoModelForCausalLM.from_pretrained("whaoyang/gemma-3-4b-novision-quant-rk3588-1.2.1", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use whaoyang/gemma-3-4b-novision-quant-rk3588-1.2.1 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "whaoyang/gemma-3-4b-novision-quant-rk3588-1.2.1"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "whaoyang/gemma-3-4b-novision-quant-rk3588-1.2.1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/whaoyang/gemma-3-4b-novision-quant-rk3588-1.2.1

SGLang

How to use whaoyang/gemma-3-4b-novision-quant-rk3588-1.2.1 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "whaoyang/gemma-3-4b-novision-quant-rk3588-1.2.1" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "whaoyang/gemma-3-4b-novision-quant-rk3588-1.2.1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "whaoyang/gemma-3-4b-novision-quant-rk3588-1.2.1" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "whaoyang/gemma-3-4b-novision-quant-rk3588-1.2.1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use whaoyang/gemma-3-4b-novision-quant-rk3588-1.2.1 with Docker Model Runner:
```
docker model run hf.co/whaoyang/gemma-3-4b-novision-quant-rk3588-1.2.1
```

This version of gemma-3-4b-novision has been converted to run on the RK3588 NPU using w8a8 quantization and with the quant_dataset.json file from this repo as the value for the dataset param of the rkllm.build() command.

This converted rkllm model differs from my other gemma-3-4b-novision-rk3588-1.2.1 model in that rkllm.build() was run with param dataset=quant_dataset.json instead of param dataset=None in the other model.

The quant_dataset.json file was generated with Rockchip's generate_data_quant.py script with the following arguments:

max_new_tokens = 448
top_k = 64
temperature = 0.7
repetition_penalty = 1.0
apply_chat_template = True

This model has been optimized with the following LoRA: NA

This model supports a max context length of 16384.

Compatible with RKLLM version: 1.2.1

Recommended `rkllm` parameters

This model runs well in limited testing with the following rkllm library paremeters:

n_keep = -1
top_k = 64
top_p = 0.95
temperature = 0.7
repeat_penalty = 1.0
frequency_penalty = 1.0
presence_penalty = 0.0
mirostat = 0
mirostat_tau = 5.0
mirostat_eta = 0.1

It is recommended to also apply a specific chat template using the following Python methods(that hook the rkllm library):

# System prompt taken from https://docs.unsloth.ai/basics/gemma-3-how-to-run-and-fine-tune#official-recommended-inference-settings
system_prompt = "<bos><start_of_turn>user\nHello!<end_of_turn>\n<start_of_turn>model\nHey there!<end_of_turn>\n<start_of_turn>user\nWhat is 1+1?<end_of_turn>\n<start_of_turn>model\n"
prompt_prefix = "<start_of_turn>user\n"
prompt_postfix = "<end_of_turn>\n<start_of_turn>model\n"
rkllm_lib.rkllm_set_chat_template(
  llm_handle, 
  ctypes.c_char_p(system_prompt.encode('utf-8')), 
  ctypes.c_char_p(prompt_prefix.encode('utf-8')), 
  ctypes.c_char_p(prompt_postfix.encode('utf-8')))