Instructions to use mradermacher/Uni-MuMER-Qwen3.5-2B-i1-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use mradermacher/Uni-MuMER-Qwen3.5-2B-i1-GGUF with Transformers:

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("mradermacher/Uni-MuMER-Qwen3.5-2B-i1-GGUF", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use mradermacher/Uni-MuMER-Qwen3.5-2B-i1-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf mradermacher/Uni-MuMER-Qwen3.5-2B-i1-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf mradermacher/Uni-MuMER-Qwen3.5-2B-i1-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf mradermacher/Uni-MuMER-Qwen3.5-2B-i1-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf mradermacher/Uni-MuMER-Qwen3.5-2B-i1-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf mradermacher/Uni-MuMER-Qwen3.5-2B-i1-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf mradermacher/Uni-MuMER-Qwen3.5-2B-i1-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf mradermacher/Uni-MuMER-Qwen3.5-2B-i1-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf mradermacher/Uni-MuMER-Qwen3.5-2B-i1-GGUF:Q4_K_M

Use Docker

docker model run hf.co/mradermacher/Uni-MuMER-Qwen3.5-2B-i1-GGUF:Q4_K_M

LM Studio
Jan
Ollama
How to use mradermacher/Uni-MuMER-Qwen3.5-2B-i1-GGUF with Ollama:
```
ollama run hf.co/mradermacher/Uni-MuMER-Qwen3.5-2B-i1-GGUF:Q4_K_M
```
Unsloth Desktop

How to use mradermacher/Uni-MuMER-Qwen3.5-2B-i1-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf mradermacher/Uni-MuMER-Qwen3.5-2B-i1-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "mradermacher/Uni-MuMER-Qwen3.5-2B-i1-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use mradermacher/Uni-MuMER-Qwen3.5-2B-i1-GGUF with Docker Model Runner:
```
docker model run hf.co/mradermacher/Uni-MuMER-Qwen3.5-2B-i1-GGUF:Q4_K_M
```

Lemonade

How to use mradermacher/Uni-MuMER-Qwen3.5-2B-i1-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull mradermacher/Uni-MuMER-Qwen3.5-2B-i1-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Uni-MuMER-Qwen3.5-2B-i1-GGUF-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use mradermacher/Uni-MuMER-Qwen3.5-2B-i1-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf mradermacher/Uni-MuMER-Qwen3.5-2B-i1-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default mradermacher/Uni-MuMER-Qwen3.5-2B-i1-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use mradermacher/Uni-MuMER-Qwen3.5-2B-i1-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf mradermacher/Uni-MuMER-Qwen3.5-2B-i1-GGUF:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "mradermacher/Uni-MuMER-Qwen3.5-2B-i1-GGUF:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

mradermacher commited on Apr 14

Commit

7ab9733

verified ·

1 Parent(s): ddb994f

uploaded from nico1

Browse files

Files changed (25) hide show

.gitattributes +24 -0
Uni-MuMER-Qwen3.5-2B.i1-IQ1_M.gguf +3 -0
Uni-MuMER-Qwen3.5-2B.i1-IQ1_S.gguf +3 -0
Uni-MuMER-Qwen3.5-2B.i1-IQ2_M.gguf +3 -0
Uni-MuMER-Qwen3.5-2B.i1-IQ2_S.gguf +3 -0
Uni-MuMER-Qwen3.5-2B.i1-IQ2_XS.gguf +3 -0
Uni-MuMER-Qwen3.5-2B.i1-IQ2_XXS.gguf +3 -0
Uni-MuMER-Qwen3.5-2B.i1-IQ3_M.gguf +3 -0
Uni-MuMER-Qwen3.5-2B.i1-IQ3_S.gguf +3 -0
Uni-MuMER-Qwen3.5-2B.i1-IQ3_XS.gguf +3 -0
Uni-MuMER-Qwen3.5-2B.i1-IQ3_XXS.gguf +3 -0
Uni-MuMER-Qwen3.5-2B.i1-IQ4_NL.gguf +3 -0
Uni-MuMER-Qwen3.5-2B.i1-IQ4_XS.gguf +3 -0
Uni-MuMER-Qwen3.5-2B.i1-Q2_K.gguf +3 -0
Uni-MuMER-Qwen3.5-2B.i1-Q2_K_S.gguf +3 -0
Uni-MuMER-Qwen3.5-2B.i1-Q3_K_L.gguf +3 -0
Uni-MuMER-Qwen3.5-2B.i1-Q3_K_M.gguf +3 -0
Uni-MuMER-Qwen3.5-2B.i1-Q3_K_S.gguf +3 -0
Uni-MuMER-Qwen3.5-2B.i1-Q4_0.gguf +3 -0
Uni-MuMER-Qwen3.5-2B.i1-Q4_1.gguf +3 -0
Uni-MuMER-Qwen3.5-2B.i1-Q4_K_M.gguf +3 -0
Uni-MuMER-Qwen3.5-2B.i1-Q4_K_S.gguf +3 -0
Uni-MuMER-Qwen3.5-2B.i1-Q5_K_M.gguf +3 -0
Uni-MuMER-Qwen3.5-2B.i1-Q5_K_S.gguf +3 -0
Uni-MuMER-Qwen3.5-2B.i1-Q6_K.gguf +3 -0

.gitattributes CHANGED Viewed

@@ -34,3 +34,27 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 Uni-MuMER-Qwen3.5-2B.imatrix.gguf filter=lfs diff=lfs merge=lfs -text

 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 Uni-MuMER-Qwen3.5-2B.imatrix.gguf filter=lfs diff=lfs merge=lfs -text
+Uni-MuMER-Qwen3.5-2B.i1-IQ1_M.gguf filter=lfs diff=lfs merge=lfs -text
+Uni-MuMER-Qwen3.5-2B.i1-IQ1_S.gguf filter=lfs diff=lfs merge=lfs -text
+Uni-MuMER-Qwen3.5-2B.i1-IQ2_M.gguf filter=lfs diff=lfs merge=lfs -text
+Uni-MuMER-Qwen3.5-2B.i1-IQ2_S.gguf filter=lfs diff=lfs merge=lfs -text
+Uni-MuMER-Qwen3.5-2B.i1-IQ2_XS.gguf filter=lfs diff=lfs merge=lfs -text
+Uni-MuMER-Qwen3.5-2B.i1-IQ2_XXS.gguf filter=lfs diff=lfs merge=lfs -text
+Uni-MuMER-Qwen3.5-2B.i1-IQ3_M.gguf filter=lfs diff=lfs merge=lfs -text
+Uni-MuMER-Qwen3.5-2B.i1-IQ3_S.gguf filter=lfs diff=lfs merge=lfs -text
+Uni-MuMER-Qwen3.5-2B.i1-IQ3_XS.gguf filter=lfs diff=lfs merge=lfs -text
+Uni-MuMER-Qwen3.5-2B.i1-IQ3_XXS.gguf filter=lfs diff=lfs merge=lfs -text
+Uni-MuMER-Qwen3.5-2B.i1-IQ4_NL.gguf filter=lfs diff=lfs merge=lfs -text
+Uni-MuMER-Qwen3.5-2B.i1-IQ4_XS.gguf filter=lfs diff=lfs merge=lfs -text
+Uni-MuMER-Qwen3.5-2B.i1-Q2_K.gguf filter=lfs diff=lfs merge=lfs -text
+Uni-MuMER-Qwen3.5-2B.i1-Q2_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+Uni-MuMER-Qwen3.5-2B.i1-Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text
+Uni-MuMER-Qwen3.5-2B.i1-Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+Uni-MuMER-Qwen3.5-2B.i1-Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+Uni-MuMER-Qwen3.5-2B.i1-Q4_0.gguf filter=lfs diff=lfs merge=lfs -text
+Uni-MuMER-Qwen3.5-2B.i1-Q4_1.gguf filter=lfs diff=lfs merge=lfs -text
+Uni-MuMER-Qwen3.5-2B.i1-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+Uni-MuMER-Qwen3.5-2B.i1-Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+Uni-MuMER-Qwen3.5-2B.i1-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+Uni-MuMER-Qwen3.5-2B.i1-Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+Uni-MuMER-Qwen3.5-2B.i1-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text

Uni-MuMER-Qwen3.5-2B.i1-IQ1_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:1ab61e36f060794f6d6b625ed43e8da1372f835d6c72e7d970d00f6127cd37e4
+size 748204928

Uni-MuMER-Qwen3.5-2B.i1-IQ1_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:5ae0101e9f3c8c1658db7fdc4aca018c6eab87a1c1d19475f03d63c09063952d
+size 722962304

Uni-MuMER-Qwen3.5-2B.i1-IQ2_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:640d721cfc7980480765a2b20fba6f7eb3087fb565b0948f832980a026d198b6
+size 865748864

Uni-MuMER-Qwen3.5-2B.i1-IQ2_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:2320312bbb405fe6dc69a88dd9f13d58f85bd2beccc2333f0f538806cdaffbd1
+size 832092032

Uni-MuMER-Qwen3.5-2B.i1-IQ2_XS.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:3740b58062bedcf69863daf349eed0e6bc6354afd61149b0a4f9651959e23ea2
+size 824719232

Uni-MuMER-Qwen3.5-2B.i1-IQ2_XXS.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:c6bec3f8466182f7a86e2b020991cb7e058fe109bbf01026855c74457706c8db
+size 790275968

Uni-MuMER-Qwen3.5-2B.i1-IQ3_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:0a3c138bd18113f0b9c194f8c299374186bc3a8274db5caf94a3267a85413db9
+size 1059446656

Uni-MuMER-Qwen3.5-2B.i1-IQ3_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:09dd2e59f8af38b2d5770215495fe28e0603bb902f5478b88235efd65fcacabe
+size 1051090816

Uni-MuMER-Qwen3.5-2B.i1-IQ3_XS.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:f9ad053f75d7b963ea55e5fbc150c74cf8b53c50e874d6c59816c1e81b01cbec
+size 1027202944

Uni-MuMER-Qwen3.5-2B.i1-IQ3_XXS.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:590f3e2446a29bffaddc95b2da83bfd5bf3d98b1a54bdbbd2d05574506e0882e
+size 927754112

Uni-MuMER-Qwen3.5-2B.i1-IQ4_NL.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:dfc962a8e8a841e7f323b516530cdbcde892c21a5385e1b17236669662a94d3a
+size 1231586176

Uni-MuMER-Qwen3.5-2B.i1-IQ4_XS.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:7ade597a2e454dc9348b18021d6e9c5414a2b159b6e85bfc85aaab0c7710aa0b
+size 1195963264

Uni-MuMER-Qwen3.5-2B.i1-Q2_K.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:369ab14d1c84757f060a380b9736d9fc1ecaed553c5c73b7a590c683623abdc6
+size 968543104

Uni-MuMER-Qwen3.5-2B.i1-Q2_K_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:dfd086fc08842257196ba71699cfa4f17e5833c533e1ba6cbd488855066a0e4b
+size 944163712

Uni-MuMER-Qwen3.5-2B.i1-Q3_K_L.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:dc03050561b98d6bb17e0aca0bd2fb309c1848b65438dbfd28bd2e9eacbfba95
+size 1164533632

Uni-MuMER-Qwen3.5-2B.i1-Q3_K_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:b133946b83634e89f50ba35e1315402c3ec1bd8cfbec6cd3d206000b40a5e022
+size 1099259776

Uni-MuMER-Qwen3.5-2B.i1-Q3_K_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:39a690b15cb8d6685c20ac4dafd2bf512b01c7646b1743b7f27c4307b3eb61a9
+size 1020174208

Uni-MuMER-Qwen3.5-2B.i1-Q4_0.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:fa9948938c522906b6d8112e5d247a0203efdf203fdb439a02e9a1a90a944b2f
+size 1204847488

Uni-MuMER-Qwen3.5-2B.i1-Q4_1.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:a7858ed703e3dd1071cf8c81d8c0575fe1278a0927ea392891ded3826357ffaa
+size 1288283008

Uni-MuMER-Qwen3.5-2B.i1-Q4_K_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:2909fd087356cd92a2f91b12866a9e662c2591bba58d47390731c2976cc29b63
+size 1274397568

Uni-MuMER-Qwen3.5-2B.i1-Q4_K_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:6bf3b4c8ab103ba2bc8ee16c290e007e69a67f4d70193188ec1f0c609d8b0c51
+size 1212056448

Uni-MuMER-Qwen3.5-2B.i1-Q5_K_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:5b549aee887fa6801257b78639cfc18d2ac7fd83b9b9067c583731c7312e10eb
+size 1411122048

Uni-MuMER-Qwen3.5-2B.i1-Q5_K_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:3f00b26eb3dab73acfc40ab7730c1f13b265d1159a8b416c6a63be78cd26cebf
+size 1374077824

Uni-MuMER-Qwen3.5-2B.i1-Q6_K.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:400ca63f41bb19091bbcc0189e5e57db65eec9277555cd072b7f02da52b420df
+size 1556391808