Instructions to use ashton25549/gemma-4-26B-A4B-it-UD-Q4_K_M-layers with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use ashton25549/gemma-4-26B-A4B-it-UD-Q4_K_M-layers with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf ashton25549/gemma-4-26B-A4B-it-UD-Q4_K_M-layers
# Run inference directly in the terminal:
llama cli -hf ashton25549/gemma-4-26B-A4B-it-UD-Q4_K_M-layers

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf ashton25549/gemma-4-26B-A4B-it-UD-Q4_K_M-layers
# Run inference directly in the terminal:
llama cli -hf ashton25549/gemma-4-26B-A4B-it-UD-Q4_K_M-layers

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf ashton25549/gemma-4-26B-A4B-it-UD-Q4_K_M-layers
# Run inference directly in the terminal:
./llama-cli -hf ashton25549/gemma-4-26B-A4B-it-UD-Q4_K_M-layers

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf ashton25549/gemma-4-26B-A4B-it-UD-Q4_K_M-layers
# Run inference directly in the terminal:
./build/bin/llama-cli -hf ashton25549/gemma-4-26B-A4B-it-UD-Q4_K_M-layers

Use Docker

docker model run hf.co/ashton25549/gemma-4-26B-A4B-it-UD-Q4_K_M-layers

LM Studio
Jan

vLLM

How to use ashton25549/gemma-4-26B-A4B-it-UD-Q4_K_M-layers with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ashton25549/gemma-4-26B-A4B-it-UD-Q4_K_M-layers"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ashton25549/gemma-4-26B-A4B-it-UD-Q4_K_M-layers",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/ashton25549/gemma-4-26B-A4B-it-UD-Q4_K_M-layers

Ollama
How to use ashton25549/gemma-4-26B-A4B-it-UD-Q4_K_M-layers with Ollama:
```
ollama run hf.co/ashton25549/gemma-4-26B-A4B-it-UD-Q4_K_M-layers
```
Unsloth Desktop

How to use ashton25549/gemma-4-26B-A4B-it-UD-Q4_K_M-layers with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf ashton25549/gemma-4-26B-A4B-it-UD-Q4_K_M-layers

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "ashton25549/gemma-4-26B-A4B-it-UD-Q4_K_M-layers"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use ashton25549/gemma-4-26B-A4B-it-UD-Q4_K_M-layers with Docker Model Runner:
```
docker model run hf.co/ashton25549/gemma-4-26B-A4B-it-UD-Q4_K_M-layers
```

Lemonade

How to use ashton25549/gemma-4-26B-A4B-it-UD-Q4_K_M-layers with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull ashton25549/gemma-4-26B-A4B-it-UD-Q4_K_M-layers

Run and chat with the model

lemonade run user.gemma-4-26B-A4B-it-UD-Q4_K_M-layers-{{QUANT_TAG}}

List all available models

lemonade list

Hermes Agent

How to use ashton25549/gemma-4-26B-A4B-it-UD-Q4_K_M-layers with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf ashton25549/gemma-4-26B-A4B-it-UD-Q4_K_M-layers

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default ashton25549/gemma-4-26B-A4B-it-UD-Q4_K_M-layers

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use ashton25549/gemma-4-26B-A4B-it-UD-Q4_K_M-layers with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf ashton25549/gemma-4-26B-A4B-it-UD-Q4_K_M-layers

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "ashton25549/gemma-4-26B-A4B-it-UD-Q4_K_M-layers" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

gemma-4-26B-A4B-it-UD-Q4_K_M-layers / model-package.json

ashton25549

jamesdumay

Duplicate from meshllm/gemma-4-26B-A4B-it-UD-Q4_K_M-layers

5b82874 3 months ago

Raw History Blame Contribute Delete

9.17 kB

	{
	"schema_version": 1,
	"model_id": "unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_M",
	"source_model": {
	"path": "/source/gemma-4-26B-A4B-it-UD-Q4_K_M.gguf",
	"sha256": "34c746b1d50ab813e29cd46c4796e3f43c741901a582f93a67b55b9fc9687b35",
	"repo": "unsloth/gemma-4-26B-A4B-it-GGUF",
	"revision": "main",
	"primary_file": "gemma-4-26B-A4B-it-UD-Q4_K_M.gguf",
	"canonical_ref": "unsloth/gemma-4-26B-A4B-it-GGUF@main/gemma-4-26B-A4B-it-UD-Q4_K_M.gguf",
	"distribution_id": "gemma-4-26B-A4B-it-UD-Q4_K_M",
	"files": [
	{
	"path": "gemma-4-26B-A4B-it-UD-Q4_K_M.gguf",
	"size_bytes": null,
	"sha256": null
	}
	]
	},
	"format": "layer-package",
	"layer_count": 30,
	"activation_width": 2816,
	"shared": {
	"metadata": {
	"path": "shared/metadata.gguf",
	"tensor_count": 1,
	"tensor_bytes": 1024,
	"artifact_bytes": 15783680,
	"sha256": "4ea990cfa36597971790aa1dcc4658cd8c946685b3d16dbaac4f685efb382002"
	},
	"embeddings": {
	"path": "shared/embeddings.gguf",
	"tensor_count": 2,
	"tensor_bytes": 784335872,
	"artifact_bytes": 800118592,
	"sha256": "6eb39617e6999290dd4c338cdcadf98ea42f948812fb4aeb0eab50735c7e0dd3"
	},
	"output": {
	"path": "shared/output.gguf",
	"tensor_count": 2,
	"tensor_bytes": 12288,
	"artifact_bytes": 15794976,
	"sha256": "88f7262cffae57f67ffb7b8db8fa9c04439d43a3af7d53fa3df81f0c6511e522"
	}
	},
	"layers": [
	{
	"layer_index": 0,
	"path": "layers/layer-000.gguf",
	"tensor_count": 23,
	"tensor_bytes": 533049860,
	"artifact_bytes": 548833888,
	"sha256": "795547711d296925216f8dbb7581e90bcf56e784fc7c86a3316c290e778825bb"
	},
	{
	"layer_index": 1,
	"path": "layers/layer-001.gguf",
	"tensor_count": 23,
	"tensor_bytes": 533049860,
	"artifact_bytes": 548833888,
	"sha256": "5fd27f0ebdf8e41606bb413c6274c5095a4c2460794f1065e60497bdbd2e83ad"
	},
	{
	"layer_index": 2,
	"path": "layers/layer-002.gguf",
	"tensor_count": 23,
	"tensor_bytes": 533049860,
	"artifact_bytes": 548833888,
	"sha256": "f1183555ce8609288aae48e2f5cd5fafe0722a0213f1ca2fe60c81b3774c2bf4"
	},
	{
	"layer_index": 3,
	"path": "layers/layer-003.gguf",
	"tensor_count": 23,
	"tensor_bytes": 533049860,
	"artifact_bytes": 548833888,
	"sha256": "6314998bdc4e7d8691b474a3f307f652aaf28a7a6490cd0d478b370b40bae8bc"
	},
	{
	"layer_index": 4,
	"path": "layers/layer-004.gguf",
	"tensor_count": 23,
	"tensor_bytes": 533049860,
	"artifact_bytes": 548833888,
	"sha256": "248eeb46a7de698b8fb3a3f819b6b2e0b0db489fb09203223aa7baa1c0944176"
	},
	{
	"layer_index": 5,
	"path": "layers/layer-005.gguf",
	"tensor_count": 22,
	"tensor_bytes": 548370948,
	"artifact_bytes": 564154912,
	"sha256": "641b96fa07b4cd828402dca6315d58558e95aee434847ef845e45ff09bfd6517"
	},
	{
	"layer_index": 6,
	"path": "layers/layer-006.gguf",
	"tensor_count": 23,
	"tensor_bytes": 533049860,
	"artifact_bytes": 548833888,
	"sha256": "d4a6f13dccc41def2309c8451f2032271437173c14dfd5095f1c6ad2a4cf5057"
	},
	{
	"layer_index": 7,
	"path": "layers/layer-007.gguf",
	"tensor_count": 23,
	"tensor_bytes": 533049860,
	"artifact_bytes": 548833888,
	"sha256": "7ba974bc7d04cf4121d773344cce620771443a3e8782a1691354568b1999dc26"
	},
	{
	"layer_index": 8,
	"path": "layers/layer-008.gguf",
	"tensor_count": 23,
	"tensor_bytes": 533049860,
	"artifact_bytes": 548833888,
	"sha256": "208176b6c2371cffa80da8c1d4ad546b6657f78c0212ca909750888dc2178f65"
	},
	{
	"layer_index": 9,
	"path": "layers/layer-009.gguf",
	"tensor_count": 23,
	"tensor_bytes": 533049860,
	"artifact_bytes": 548833888,
	"sha256": "1b6a5ebd9996ad7845eb2013afeb4eee5bd137225b1b62ac6c681f3d9ecb39a0"
	},
	{
	"layer_index": 10,
	"path": "layers/layer-010.gguf",
	"tensor_count": 23,
	"tensor_bytes": 533049860,
	"artifact_bytes": 548833888,
	"sha256": "c6be8c6b489a4cbb41c1d4df9e634a63e8a617f4bc833546aa69557dab88f406"
	},
	{
	"layer_index": 11,
	"path": "layers/layer-011.gguf",
	"tensor_count": 22,
	"tensor_bytes": 548370948,
	"artifact_bytes": 564154912,
	"sha256": "549aebbb82c7021df8ee3e5118666f9438bb8274b6b1c67ad1debdbe5bad8979"
	},
	{
	"layer_index": 12,
	"path": "layers/layer-012.gguf",
	"tensor_count": 23,
	"tensor_bytes": 533049860,
	"artifact_bytes": 548833888,
	"sha256": "1de53cfc0164d4ada8cf3dac92f26ee10d3de807ebc282b098dffd6640a3d3c0"
	},
	{
	"layer_index": 13,
	"path": "layers/layer-013.gguf",
	"tensor_count": 23,
	"tensor_bytes": 533049860,
	"artifact_bytes": 548833888,
	"sha256": "d316d50fdd561caa794f9ff3c06f01e3a9448c864bc4e5f0efc7166091b8be45"
	},
	{
	"layer_index": 14,
	"path": "layers/layer-014.gguf",
	"tensor_count": 23,
	"tensor_bytes": 533049860,
	"artifact_bytes": 548833888,
	"sha256": "d318ef09a4b1efe146378ebb6a1c6b6fa7a88c95a208ef72e1f32c531851dd52"
	},
	{
	"layer_index": 15,
	"path": "layers/layer-015.gguf",
	"tensor_count": 23,
	"tensor_bytes": 533049860,
	"artifact_bytes": 548833888,
	"sha256": "3097d923048c9ac2cb61ee67157d22e21d57cfdca59a5ba68c4cbd9954eb8f9e"
	},
	{
	"layer_index": 16,
	"path": "layers/layer-016.gguf",
	"tensor_count": 23,
	"tensor_bytes": 533049860,
	"artifact_bytes": 548833888,
	"sha256": "24e34c8798c1e3e9e09ac84de7ae9f95c10c42304d753d924f6a8531bfcc027f"
	},
	{
	"layer_index": 17,
	"path": "layers/layer-017.gguf",
	"tensor_count": 22,
	"tensor_bytes": 548370948,
	"artifact_bytes": 564154912,
	"sha256": "4bb63596cb3430ba77c254e92292f82f8c71db009c32d9ba584fc45af12f9af8"
	},
	{
	"layer_index": 18,
	"path": "layers/layer-018.gguf",
	"tensor_count": 23,
	"tensor_bytes": 533049860,
	"artifact_bytes": 548833888,
	"sha256": "f415d2702a183a04498db056a389d83a42454378989ab02b72c8b07475cb528f"
	},
	{
	"layer_index": 19,
	"path": "layers/layer-019.gguf",
	"tensor_count": 23,
	"tensor_bytes": 533049860,
	"artifact_bytes": 548833888,
	"sha256": "ff78b0c3c7d08389a45838c63047098e2d42de6dcad4cee5195647a95d3fa7f8"
	},
	{
	"layer_index": 20,
	"path": "layers/layer-020.gguf",
	"tensor_count": 23,
	"tensor_bytes": 533049860,
	"artifact_bytes": 548833888,
	"sha256": "60976a6462e78f8b76693fcf9e97936a544cdb5afbbc72dff4cf6bacdbf724f0"
	},
	{
	"layer_index": 21,
	"path": "layers/layer-021.gguf",
	"tensor_count": 23,
	"tensor_bytes": 533049860,
	"artifact_bytes": 548833888,
	"sha256": "0a68b6df3c5622e784d46e2a1a1154f2b4ec454cb226181134bb9106fef17207"
	},
	{
	"layer_index": 22,
	"path": "layers/layer-022.gguf",
	"tensor_count": 23,
	"tensor_bytes": 533049860,
	"artifact_bytes": 548833888,
	"sha256": "c9b688bc005207a9cadc2c980e2a781511177ecc79cb3eca050b8388f0183b31"
	},
	{
	"layer_index": 23,
	"path": "layers/layer-023.gguf",
	"tensor_count": 22,
	"tensor_bytes": 548370948,
	"artifact_bytes": 564154912,
	"sha256": "bbc793dcf243eb8a4dd2185d1b0d55751de919e97ae534341defd79a08e807de"
	},
	{
	"layer_index": 24,
	"path": "layers/layer-024.gguf",
	"tensor_count": 23,
	"tensor_bytes": 533049860,
	"artifact_bytes": 548833888,
	"sha256": "f2af485e03abb1e19079f2be5cdd179dda3757ff6ee9256cad5119f894b44f1b"
	},
	{
	"layer_index": 25,
	"path": "layers/layer-025.gguf",
	"tensor_count": 23,
	"tensor_bytes": 533049860,
	"artifact_bytes": 548833888,
	"sha256": "0b98eca1e6326417a48e4beba4a051f6b575f42ba528d5555e5f301e68bc702f"
	},
	{
	"layer_index": 26,
	"path": "layers/layer-026.gguf",
	"tensor_count": 23,
	"tensor_bytes": 533049860,
	"artifact_bytes": 548833888,
	"sha256": "c293849aedfe2283364f9307473156e3c0fc52db3fc2632e14416ec85042b458"
	},
	{
	"layer_index": 27,
	"path": "layers/layer-027.gguf",
	"tensor_count": 23,
	"tensor_bytes": 533049860,
	"artifact_bytes": 548833888,
	"sha256": "0b793d8520461133a443f82c490cae8066ffec1866a98b24ded842ffff934ff9"
	},
	{
	"layer_index": 28,
	"path": "layers/layer-028.gguf",
	"tensor_count": 23,
	"tensor_bytes": 533049860,
	"artifact_bytes": 548833888,
	"sha256": "ccbff59f737371d43fbf6c084e0a1f11c01533804c2a039713f4e0a6c153020b"
	},
	{
	"layer_index": 29,
	"path": "layers/layer-029.gguf",
	"tensor_count": 22,
	"tensor_bytes": 627669508,
	"artifact_bytes": 643453472,
	"sha256": "089d9e4a87326e2791ebfb7b61275ab92f542afec731deced1a796856693efb2"
	}
	],
	"skippy_abi_version": "0.1.22",
	"created_at_unix_secs": 1778213236
	}