Instructions to use unsloth/gemma-4-26B-A4B-it-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use unsloth/gemma-4-26B-A4B-it-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_M
# Run inference directly in the terminal:
llama cli -hf unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_M
# Run inference directly in the terminal:
llama cli -hf unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_M

Use Docker

docker model run hf.co/unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_M

LM Studio
Jan

vLLM

How to use unsloth/gemma-4-26B-A4B-it-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "unsloth/gemma-4-26B-A4B-it-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "unsloth/gemma-4-26B-A4B-it-GGUF",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_M

Ollama
How to use unsloth/gemma-4-26B-A4B-it-GGUF with Ollama:
```
ollama run hf.co/unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_M
```
Unsloth Desktop

How to use unsloth/gemma-4-26B-A4B-it-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use unsloth/gemma-4-26B-A4B-it-GGUF with Docker Model Runner:
```
docker model run hf.co/unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_M
```

Lemonade

How to use unsloth/gemma-4-26B-A4B-it-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_M

Run and chat with the model

lemonade run user.gemma-4-26B-A4B-it-GGUF-UD-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use unsloth/gemma-4-26B-A4B-it-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use unsloth/gemma-4-26B-A4B-it-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

danielhanchen commited on May 4

Commit

b19ae87

verified ·

1 Parent(s): b68961b

New Gemma chat template update by Google

Browse files

Files changed (25) hide show

BF16/gemma-4-26B-A4B-it-BF16-00001-of-00002.gguf +2 -2
gemma-4-26B-A4B-it-MXFP4_MOE.gguf +2 -2
gemma-4-26B-A4B-it-Q8_0.gguf +2 -2
gemma-4-26B-A4B-it-UD-IQ2_M.gguf +2 -2
gemma-4-26B-A4B-it-UD-IQ2_XXS.gguf +2 -2
gemma-4-26B-A4B-it-UD-IQ3_S.gguf +2 -2
gemma-4-26B-A4B-it-UD-IQ3_XXS.gguf +2 -2
gemma-4-26B-A4B-it-UD-IQ4_NL.gguf +2 -2
gemma-4-26B-A4B-it-UD-IQ4_XS.gguf +2 -2
gemma-4-26B-A4B-it-UD-Q2_K_XL.gguf +2 -2
gemma-4-26B-A4B-it-UD-Q3_K_M.gguf +2 -2
gemma-4-26B-A4B-it-UD-Q3_K_XL.gguf +2 -2
gemma-4-26B-A4B-it-UD-Q4_K_M.gguf +2 -2
gemma-4-26B-A4B-it-UD-Q4_K_S.gguf +2 -2
gemma-4-26B-A4B-it-UD-Q4_K_XL.gguf +2 -2
gemma-4-26B-A4B-it-UD-Q5_K_M.gguf +2 -2
gemma-4-26B-A4B-it-UD-Q5_K_S.gguf +2 -2
gemma-4-26B-A4B-it-UD-Q5_K_XL.gguf +2 -2
gemma-4-26B-A4B-it-UD-Q6_K.gguf +2 -2
gemma-4-26B-A4B-it-UD-Q6_K_XL.gguf +2 -2
gemma-4-26B-A4B-it-UD-Q8_K_XL.gguf +2 -2
imatrix_unsloth.gguf_file +1 -1
mmproj-BF16.gguf +2 -2
mmproj-F16.gguf +2 -2
mmproj-F32.gguf +2 -2

BF16/gemma-4-26B-A4B-it-BF16-00001-of-00002.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:39822285661ea71080dc15832e6603c06ce5c19613e20d59ebe108de1d0b2c62
-size 49923213088

 version https://git-lfs.github.com/spec/v1
+oid sha256:e4f5879330c54468ca1d1620291cfc38e0c33201c19d4fb34f3cecc5d7fe6e44
+size 49923213568

gemma-4-26B-A4B-it-MXFP4_MOE.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ba25642c352fda86a38ae47114b9603e8ec1fb527e0a2ddf458001fcc53aa866
-size 16630345024

 version https://git-lfs.github.com/spec/v1
+oid sha256:4599b52f6b74ae64ccfe1424695cb5ba85e672045175bcd7c8cab46b2538f3e0
+size 16551046944

gemma-4-26B-A4B-it-Q8_0.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:af5f307a0d72b99be5fa547573ea161dafc7164467f33601598a6e64a43f6d80
-size 26859859264

 version https://git-lfs.github.com/spec/v1
+oid sha256:b26c56ea4bf724b4efa0161bb4615e974847ec64450a4dd49da8712614a128c7
+size 26859859744

gemma-4-26B-A4B-it-UD-IQ2_M.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:394f5074fc5d07af610c5bec113ac2979de542d80da20e290ce70bf0572ef7b0
-size 9974943040

 version https://git-lfs.github.com/spec/v1
+oid sha256:9999bd11e44ea81c93d1652e45631d531963b2b24eae6ef7bf7b524cdaf79ed0
+size 10014753312

gemma-4-26B-A4B-it-UD-IQ2_XXS.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:bd1866a6ae011422534d53f78a259d3f4ffd24f02e112e2c2e8dbd49aafa32a9
-size 9882668352

 version https://git-lfs.github.com/spec/v1
+oid sha256:f1bd775a37603e36e2f96d799072909cd0245f1509945424e0bece4111ac90c7
+size 9922478624

gemma-4-26B-A4B-it-UD-IQ3_S.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:2f8dc9c2c7756e115f10f38983f5369df036ac24a82c09c668014bea249b1b44
-size 11219406656

 version https://git-lfs.github.com/spec/v1
+oid sha256:f164ce43ae5968d2d7ef7f3a8189960ea2172242ba1588b026f19754b8034bb1
+size 11289669152

gemma-4-26B-A4B-it-UD-IQ3_XXS.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3e54bc50c8a25d4e761abaeba13205a7956b3d9d1f7b10482adc6156c5c5276b
-size 11219406656

 version https://git-lfs.github.com/spec/v1
+oid sha256:c2fefe93b9bb964ba5433bf38a907c054adc33d4bd7d044c4c0bdb5963a970b4
+size 11416546848

gemma-4-26B-A4B-it-UD-IQ4_NL.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0fa0c968cda1c34f95a53dad4ebfc1a7c12628ca38185f352dbe043436064343
-size 13418753344

 version https://git-lfs.github.com/spec/v1
+oid sha256:ae99d08812477214799e20624c39c86fc681423ae673fa84553eb14a3b8e449b
+size 13613035296

gemma-4-26B-A4B-it-UD-IQ4_XS.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ab05e3504fae607c205dae0117d58e3fddf0e88f968a0fbf307437c68398bad9
-size 13418753344

 version https://git-lfs.github.com/spec/v1
+oid sha256:694a7d2a3b994655db30c8eff7f77933357efed36c6551afb68d14a07484fde5
+size 13597175584

gemma-4-26B-A4B-it-UD-Q2_K_XL.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:1e3a85588478801c24759be68f139ae6dff9a1f092ff06ede0f4b9a4ad5a5cb3
-size 10545963072

 version https://git-lfs.github.com/spec/v1
+oid sha256:27253964b2e935923a4e9eb435033d39a84cdb75c2185c4031ff9e4559674e6f
+size 10546932256

gemma-4-26B-A4B-it-UD-Q3_K_M.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:6296b39d581668c4b654617d6d49c9edca15635fb803c3c9f6d813771df85d1e
-size 12526284096

 version https://git-lfs.github.com/spec/v1
+oid sha256:985f740c7b928d15ecd8bf2bf4506d12ae6080c488f11eb8a4a97b215274bf7d
+size 12728495904

gemma-4-26B-A4B-it-UD-Q3_K_XL.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:5798a39ad0b64bb054c8ef6c0885d089885a948a82da21424219b6935c18b9f6
-size 12875558208

 version https://git-lfs.github.com/spec/v1
+oid sha256:1cb1c2349a76e97945276b8f9dabd92d334bdf4ca3290279287e85421ea65985
+size 12907278112

gemma-4-26B-A4B-it-UD-Q4_K_M.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b8707e57f676d8dd1b80f623b45200cc92e6966b0e95275e606f412095a49fde
-size 16868240704

 version https://git-lfs.github.com/spec/v1
+oid sha256:34c746b1d50ab813e29cd46c4796e3f43c741901a582f93a67b55b9fc9687b35
+size 16947539744

gemma-4-26B-A4B-it-UD-Q4_K_S.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c0d205a16df9359b8d97e8c53824a0c4200d3d8e2a0c7629065f15ca78b708e7
-size 16392449344

 version https://git-lfs.github.com/spec/v1
+oid sha256:395c4e5620ca01544a0a4a15715a78b753115fac5a521ceeda3497dd9c481fc4
+size 16487608096

gemma-4-26B-A4B-it-UD-Q4_K_XL.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:87121467dee16160e5678be823187079784ccda8f20d3ad2f77a77739581a148
-size 17090276672

 version https://git-lfs.github.com/spec/v1
+oid sha256:453cf049ba87a29b9ed5739087b84b7fa0265a4f2b11eefa2c77683dec6a8020
+size 17010978592

gemma-4-26B-A4B-it-UD-Q5_K_M.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f19acf0c0329932bc8e66ee904398140569658ccb65075182f198b3b07edcf7f
-size 21150362944

 version https://git-lfs.github.com/spec/v1
+oid sha256:f2fe28fc1d82e7c74f47d570a8c8847513fe2712a1b3a5bcd869031d952c4936
+size 21150363424

gemma-4-26B-A4B-it-UD-Q5_K_S.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:8645e8668f98d13d1d9bb58e2d3672fb4debb1ead401ca3adc12a6ddb442426f
-size 18771406144

 version https://git-lfs.github.com/spec/v1
+oid sha256:58128b2c82d858120b71982f27119a9ca276c8eb46276e1da257e6eca9d90897
+size 18850705184

gemma-4-26B-A4B-it-UD-Q5_K_XL.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ffb15f755bcda2dced5524490ce8a4106e557879b3e6f1a1d0d225016205bd64
-size 21217766720

 version https://git-lfs.github.com/spec/v1
+oid sha256:ca7df7d29d2abeb38957bd7f2ed0e7124c0a04337a453c2d19d7b88dfb067214
+size 21217767200

gemma-4-26B-A4B-it-UD-Q6_K.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:59ac620521cd8519ad4720f09ea58bce4b9f0c35684f5c062666cd6562c82f31
-size 23172476224

 version https://git-lfs.github.com/spec/v1
+oid sha256:d3d9e6a63845bdc83e9f9fc5923e77c023ccc1197c9e145e6a8754bad80b5d75
+size 23172476704

gemma-4-26B-A4B-it-UD-Q6_K_XL.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:8a8697b42b455ace3853bd5eca725a8a071757f3ae0678cab932918211e394db
-size 23295388992

 version https://git-lfs.github.com/spec/v1
+oid sha256:5cfb7ab424c01388538005f26573f3bd374d3140cc021a1c44249e69928882a4
+size 23295389472

gemma-4-26B-A4B-it-UD-Q8_K_XL.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:9c961effefe478b33e4f4f63ba225aa51b3ab11aa3d68fe2ad34ce4f082a241b
-size 27866185024

 version https://git-lfs.github.com/spec/v1
+oid sha256:50e180d69641e017d7e08a6f602988effde8232ff6bc0231e839636fdcc03d8f
+size 27636230944

imatrix_unsloth.gguf_file CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:6186d77efe82b850b1c52dbd408f722e8bcad0258485191d23655bdb14356f10
 size 56941536

 version https://git-lfs.github.com/spec/v1
+oid sha256:8065033ecc77d819b6bca2fe1ae5e0b0e63b2ab2cb1221663a7d7265ee7f28e9
 size 56941536

mmproj-BF16.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:fc2ebf4c44528daa2cea7b39891712847ca5e4f87dcf578054a06c46bfe6da27
-size 1194828384

 version https://git-lfs.github.com/spec/v1
+oid sha256:41926ed5f1403cf5add23b0684992805ea6f97253096132e769e65646b8cef9d
+size 1194828256

mmproj-F16.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:90a77a586ccdbabe2b5d28d9da0b61271bdb426c04f723bd01786992f00d317e
-size 1193058912

 version https://git-lfs.github.com/spec/v1
+oid sha256:418a6d8723067cd712235facbbc5cba6c8fbbd413fc1292d2aace5a027d5a42f
+size 1193058784

mmproj-F32.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ce12ca17e5f479ff292cd66817960e4d3f1b09671f744e415c98a55d7725c9ed
-size 2291200608

 version https://git-lfs.github.com/spec/v1
+oid sha256:ec31640a1f68fd7883e3ef7ef1afdc98d8b42867ff49ea16649a000447bcf163
+size 2291200480