Instructions to use AtomicChat/gemma-4-E4B-it-assistant-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use AtomicChat/gemma-4-E4B-it-assistant-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf AtomicChat/gemma-4-E4B-it-assistant-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf AtomicChat/gemma-4-E4B-it-assistant-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf AtomicChat/gemma-4-E4B-it-assistant-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf AtomicChat/gemma-4-E4B-it-assistant-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf AtomicChat/gemma-4-E4B-it-assistant-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf AtomicChat/gemma-4-E4B-it-assistant-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf AtomicChat/gemma-4-E4B-it-assistant-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf AtomicChat/gemma-4-E4B-it-assistant-GGUF:Q4_K_M

Use Docker

docker model run hf.co/AtomicChat/gemma-4-E4B-it-assistant-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use AtomicChat/gemma-4-E4B-it-assistant-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "AtomicChat/gemma-4-E4B-it-assistant-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AtomicChat/gemma-4-E4B-it-assistant-GGUF",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/AtomicChat/gemma-4-E4B-it-assistant-GGUF:Q4_K_M

Ollama
How to use AtomicChat/gemma-4-E4B-it-assistant-GGUF with Ollama:
```
ollama run hf.co/AtomicChat/gemma-4-E4B-it-assistant-GGUF:Q4_K_M
```
Unsloth Desktop
Docker Model Runner
How to use AtomicChat/gemma-4-E4B-it-assistant-GGUF with Docker Model Runner:
```
docker model run hf.co/AtomicChat/gemma-4-E4B-it-assistant-GGUF:Q4_K_M
```

Lemonade

How to use AtomicChat/gemma-4-E4B-it-assistant-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull AtomicChat/gemma-4-E4B-it-assistant-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.gemma-4-E4B-it-assistant-GGUF-Q4_K_M

List all available models

lemonade list

Atomic Chat

Server crashing on first prompt

by belx-dev - opened May 16

Discussion

belx-dev

May 16

•

edited May 17

I've succesfully used assistant models with both the dense and MOE models on RTX 4090 (win) with meaningful speed up. However, the E4B drafter keeps crashing on me. The server starts successfully but then immediately crashes after receiving first prompt.

I'm quite new to this, so I'm probably missing something obvious.

belx-dev

May 17

FIXED:
Not sure, whether the problem was in my local branch, but Claude has fixed for me after several trial and errors. The problem was in ggml/src/ggml-cuda/fattn.cu, ggml/src/ggml-cuda/fattn-mma-f16.cuh, ggml/src/ggml-cuda/template-instances/fattn-mma-f16-instance-ncols1_*-ncols2_2.cu. When flash attention MMA is used with head dimension D=512 and a GQA ratio of exactly 2 (i.e. n_head / n_head_kv == 2), the server crashes unconditionally with fatal error.

sandichhuu

Jun 6

I got crash also.
This is log:

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment