Instructions to use GuminiResearch/Gumini-1.5B-Base-i1-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use GuminiResearch/Gumini-1.5B-Base-i1-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf GuminiResearch/Gumini-1.5B-Base-i1-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf GuminiResearch/Gumini-1.5B-Base-i1-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf GuminiResearch/Gumini-1.5B-Base-i1-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf GuminiResearch/Gumini-1.5B-Base-i1-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf GuminiResearch/Gumini-1.5B-Base-i1-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf GuminiResearch/Gumini-1.5B-Base-i1-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf GuminiResearch/Gumini-1.5B-Base-i1-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf GuminiResearch/Gumini-1.5B-Base-i1-GGUF:Q4_K_M

Use Docker

docker model run hf.co/GuminiResearch/Gumini-1.5B-Base-i1-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use GuminiResearch/Gumini-1.5B-Base-i1-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "GuminiResearch/Gumini-1.5B-Base-i1-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "GuminiResearch/Gumini-1.5B-Base-i1-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/GuminiResearch/Gumini-1.5B-Base-i1-GGUF:Q4_K_M

Ollama
How to use GuminiResearch/Gumini-1.5B-Base-i1-GGUF with Ollama:
```
ollama run hf.co/GuminiResearch/Gumini-1.5B-Base-i1-GGUF:Q4_K_M
```
Unsloth Desktop

How to use GuminiResearch/Gumini-1.5B-Base-i1-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf GuminiResearch/Gumini-1.5B-Base-i1-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "GuminiResearch/Gumini-1.5B-Base-i1-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use GuminiResearch/Gumini-1.5B-Base-i1-GGUF with Docker Model Runner:
```
docker model run hf.co/GuminiResearch/Gumini-1.5B-Base-i1-GGUF:Q4_K_M
```

Lemonade

How to use GuminiResearch/Gumini-1.5B-Base-i1-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull GuminiResearch/Gumini-1.5B-Base-i1-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Gumini-1.5B-Base-i1-GGUF-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use GuminiResearch/Gumini-1.5B-Base-i1-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf GuminiResearch/Gumini-1.5B-Base-i1-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default GuminiResearch/Gumini-1.5B-Base-i1-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use GuminiResearch/Gumini-1.5B-Base-i1-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf GuminiResearch/Gumini-1.5B-Base-i1-GGUF:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "GuminiResearch/Gumini-1.5B-Base-i1-GGUF:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

🐻 Gumini-1.5B-Base-i1-GGUF (구미니)

Built with Qwen

Model Description

GGUF quantized versions of GuminiResearch/Gumini-1.5B-Base for use with llama.cpp and compatible tools (Ollama, LM Studio, etc.).

All quantizations were created using importance matrix (imatrix) calibration for optimal quality preservation.

This is a BASE model, not instruction-tuned.
It produces text continuations rather than conversational responses.

Model Details

Attribute	Value
Original Model	Gumini-1.5B-Base
Quantized by	Gumin Kwon (권구민)
Parameters	1.54B
Layers	16
Hidden Size	2048
Base PPL (F16)	8.48

Quantization Results

Perplexity Comparison

PPL vs Size Trade-off

Recommended Quantizations

Quant	PPL	Size	PPL Δ	Quality	Use Case
Q8_0	8.50	1.5G	+0.02	Excellent	Maximum quality
Q6_K	8.52	1.2G	+0.04	Excellent	High quality
Q5_K_M	8.61	1.1G	+0.13	Excellent	Balanced (recommended)
Q4_K_M	8.72	956M	+0.24	Very Good	Size optimized
IQ4_XS	8.79	876M	+0.31	Very Good	imatrix 4-bit
IQ3_M	9.09	770M	+0.61	Good	Mobile/Edge

All Quantization Results

Comparison: 1B vs 1.5B

Model	Layers	Params	PPL (F16)	Improvement
Gumini 1B	10	1.08B	15.36	-
Gumini 1.5B	16	1.54B	8.48	45% better

The 1.5B model shows significant quality improvement with only 6 additional layers!

Usage

With llama.cpp

# Download
huggingface-cli download GuminiResearch/Gumini-1.5B-Base-i1-GGUF Gumini-1.5B-Base.i1-Q4_K_M.gguf

# Run
./llama-cli -m Gumini-1.5B-Base.i1-Q4_K_M.gguf -p "저는 구미니입니다." -n 100

With Ollama

echo 'FROM ./Gumini-1.5B-Base.i1-Q4_K_M.gguf' > Modelfile
ollama create gumini-1.5b -f Modelfile
ollama run gumini-1.5b

With LM Studio

Download any .gguf file from this repo
Import into LM Studio
Start generating!

Quantization Guide

Tips

Best quality: Use Q8_0 or Q6_K
Balanced: Use Q5_K_M or Q4_K_M
Mobile/Edge: Use IQ4_XS or IQ3_M
"i1" prefix: Indicates imatrix was used during quantization

Original Model

Gumini-1.5B (구미니) is a bilingual Korean-English base language model trained using the Inheritune methodology. Starting from Qwen 2.5 3B, the model progressively grew from 10 to 16 layers through 7 training stages.

Inheritune Progressive Layer Growing

Stage 0: 10 layers (1.08B) → 393M tokens
Stage 1: 11 layers (1.15B) → 393M tokens
Stage 2: 12 layers (1.23B) → 393M tokens
Stage 3: 13 layers (1.31B) → 393M tokens
Stage 4: 14 layers (1.39B) → 393M tokens
Stage 5: 15 layers (1.47B) → 393M tokens
Stage 6: 16 layers (1.54B) → 786M tokens ⭐
────────────────────────────────────────────
Total: 16 layers, 1.54B params, ~3.14B tokens

Training Data: 80% Korean, 20% English

See GuminiResearch/Gumini-1.5B-Base for full details.

License

Qwen Research License (Non-Commercial)

This model is Built with Qwen and derived from Qwen 2.5 3B.

Qwen is licensed under the Qwen RESEARCH LICENSE AGREEMENT.
Copyright (c) Alibaba Cloud. All Rights Reserved.

This model is for NON-COMMERCIAL / RESEARCH use only.
For commercial use, contact Alibaba Cloud.

References

Inheritune Paper

@inproceedings{Sanyal2024inheritune,
  title={Inheritune: Training Smaller Yet More Attentive Language Models},
  author={Sunny Sanyal and Ravid Shwartz-Ziv and Alexandros G. Dimakis and Sujay Sanghavi},
  year={2024},
  url={https://arxiv.org/abs/2404.08634}
}

Qwen 2.5

@misc{qwen2.5,
  title={Qwen2.5: A Party of Foundation Models},
  author={Qwen Team},
  year={2024},
  url={https://qwenlm.github.io/blog/qwen2.5/}
}

Citation

@misc{gumini2025,
  title={Gumini-1.5B: Bilingual Korean-English Language Model via Inheritune},
  author={Gumin Kwon},
  year={2025},
  note={Built with Qwen. Trained with Inheritune progressive layer growing.},
  url={https://huggingface.co/GuminiResearch/Gumini-1.5B-Base-i1-GGUF}
}