Instructions to use mradermacher/zen3-reranker-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use mradermacher/zen3-reranker-GGUF with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-classification", model="mradermacher/zen3-reranker-GGUF")

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("mradermacher/zen3-reranker-GGUF", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use mradermacher/zen3-reranker-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf mradermacher/zen3-reranker-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf mradermacher/zen3-reranker-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf mradermacher/zen3-reranker-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf mradermacher/zen3-reranker-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf mradermacher/zen3-reranker-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf mradermacher/zen3-reranker-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf mradermacher/zen3-reranker-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf mradermacher/zen3-reranker-GGUF:Q4_K_M

Use Docker

docker model run hf.co/mradermacher/zen3-reranker-GGUF:Q4_K_M

LM Studio
Jan
Ollama
How to use mradermacher/zen3-reranker-GGUF with Ollama:
```
ollama run hf.co/mradermacher/zen3-reranker-GGUF:Q4_K_M
```
Unsloth Desktop
Docker Model Runner
How to use mradermacher/zen3-reranker-GGUF with Docker Model Runner:
```
docker model run hf.co/mradermacher/zen3-reranker-GGUF:Q4_K_M
```

Lemonade

How to use mradermacher/zen3-reranker-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull mradermacher/zen3-reranker-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.zen3-reranker-GGUF-Q4_K_M

List all available models

lemonade list

Atomic Chat

mradermacher commited on Feb 24

Commit

8a0810f

verified ·

1 Parent(s): 3cf3bca

uploaded from leia

Browse files

Files changed (13) hide show

.gitattributes +12 -0
zen3-reranker.IQ4_XS.gguf +3 -0
zen3-reranker.Q2_K.gguf +3 -0
zen3-reranker.Q3_K_L.gguf +3 -0
zen3-reranker.Q3_K_M.gguf +3 -0
zen3-reranker.Q3_K_S.gguf +3 -0
zen3-reranker.Q4_K_M.gguf +3 -0
zen3-reranker.Q4_K_S.gguf +3 -0
zen3-reranker.Q5_K_M.gguf +3 -0
zen3-reranker.Q5_K_S.gguf +3 -0
zen3-reranker.Q6_K.gguf +3 -0
zen3-reranker.Q8_0.gguf +3 -0
zen3-reranker.f16.gguf +3 -0

.gitattributes CHANGED Viewed

@@ -33,3 +33,15 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text

 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+zen3-reranker.IQ4_XS.gguf filter=lfs diff=lfs merge=lfs -text
+zen3-reranker.Q2_K.gguf filter=lfs diff=lfs merge=lfs -text
+zen3-reranker.Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text
+zen3-reranker.Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+zen3-reranker.Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+zen3-reranker.Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+zen3-reranker.Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+zen3-reranker.Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+zen3-reranker.Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+zen3-reranker.Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
+zen3-reranker.Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
+zen3-reranker.f16.gguf filter=lfs diff=lfs merge=lfs -text

zen3-reranker.IQ4_XS.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:9e24e851b305ee84115913ac1136ddac80b2d5c19c4b9cb81e315457ee69daae
+size 414653312

zen3-reranker.Q2_K.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:adedc213a3122e6dd45aa18c8564f7d14408632d8656d390b2297e6789e2cc25
+size 366467968

zen3-reranker.Q3_K_L.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:10cddb05ce1d45704845ad394de022eaf2c889c88c405c61f7f8c85ac0ff1f9a
+size 420838272

zen3-reranker.Q3_K_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:a386032df7aa2dd69f4bd1bfe08a413c775fe498b03a0a47197a3d4779cf75ce
+size 402750336

zen3-reranker.Q3_K_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:bb3427ab200136e8d34d9a09aee62febe89380b0ea73afa72d9c4555ba3037d3
+size 381909888

zen3-reranker.Q4_K_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:4372444453830a154e3ca7673e18cffa3a5617818cf24b91338978f0108ab35a
+size 438377344

zen3-reranker.Q4_K_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:76dcde6b950c7c6001febdcb8e895a7d163c82e3bf0ea0adc029acb753550ff0
+size 424254336

zen3-reranker.Q5_K_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:428ad5de5ae933add7b05813dc3e6dc272d35d7e299df9dbfde8c5b8cdbca247
+size 468392832

zen3-reranker.Q5_K_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:34562682048c4c6ae7b9d44c16851c0caac1924c61a98f706f6a277781fcf0ce
+size 460036992

zen3-reranker.Q6_K.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:2144cadb416f09792a2efa154d047982caa7fc1ceead49a2106920b913e25c61
+size 500284288

zen3-reranker.Q8_0.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:02a5f161d646b7e36fa3c0f72dd079ee1b4ebaecf1bf365a25831d7292860404
+size 635676896

zen3-reranker.f16.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:b1902966abbad034be94bc39629093997aaf41c3fff9c64d6128e6d17970276e
+size 1159777376