Instructions to use mradermacher/bella-bartender-heretic-3b-i1-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use mradermacher/bella-bartender-heretic-3b-i1-GGUF with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="mradermacher/bella-bartender-heretic-3b-i1-GGUF")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("mradermacher/bella-bartender-heretic-3b-i1-GGUF", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use mradermacher/bella-bartender-heretic-3b-i1-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf mradermacher/bella-bartender-heretic-3b-i1-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf mradermacher/bella-bartender-heretic-3b-i1-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf mradermacher/bella-bartender-heretic-3b-i1-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf mradermacher/bella-bartender-heretic-3b-i1-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf mradermacher/bella-bartender-heretic-3b-i1-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf mradermacher/bella-bartender-heretic-3b-i1-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf mradermacher/bella-bartender-heretic-3b-i1-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf mradermacher/bella-bartender-heretic-3b-i1-GGUF:Q4_K_M

Use Docker

docker model run hf.co/mradermacher/bella-bartender-heretic-3b-i1-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use mradermacher/bella-bartender-heretic-3b-i1-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "mradermacher/bella-bartender-heretic-3b-i1-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mradermacher/bella-bartender-heretic-3b-i1-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/mradermacher/bella-bartender-heretic-3b-i1-GGUF:Q4_K_M

SGLang

How to use mradermacher/bella-bartender-heretic-3b-i1-GGUF with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "mradermacher/bella-bartender-heretic-3b-i1-GGUF" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mradermacher/bella-bartender-heretic-3b-i1-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "mradermacher/bella-bartender-heretic-3b-i1-GGUF" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mradermacher/bella-bartender-heretic-3b-i1-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Ollama
How to use mradermacher/bella-bartender-heretic-3b-i1-GGUF with Ollama:
```
ollama run hf.co/mradermacher/bella-bartender-heretic-3b-i1-GGUF:Q4_K_M
```
Unsloth Desktop
Docker Model Runner
How to use mradermacher/bella-bartender-heretic-3b-i1-GGUF with Docker Model Runner:
```
docker model run hf.co/mradermacher/bella-bartender-heretic-3b-i1-GGUF:Q4_K_M
```

Lemonade

How to use mradermacher/bella-bartender-heretic-3b-i1-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull mradermacher/bella-bartender-heretic-3b-i1-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.bella-bartender-heretic-3b-i1-GGUF-Q4_K_M

List all available models

lemonade list

Atomic Chat

mradermacher commited on Mar 21

Commit

70331ee

verified ·

1 Parent(s): e6786b8

uploaded from marco

Browse files

Files changed (25) hide show

.gitattributes +24 -0
bella-bartender-heretic-3b.i1-IQ1_M.gguf +3 -0
bella-bartender-heretic-3b.i1-IQ1_S.gguf +3 -0
bella-bartender-heretic-3b.i1-IQ2_M.gguf +3 -0
bella-bartender-heretic-3b.i1-IQ2_S.gguf +3 -0
bella-bartender-heretic-3b.i1-IQ2_XS.gguf +3 -0
bella-bartender-heretic-3b.i1-IQ2_XXS.gguf +3 -0
bella-bartender-heretic-3b.i1-IQ3_M.gguf +3 -0
bella-bartender-heretic-3b.i1-IQ3_S.gguf +3 -0
bella-bartender-heretic-3b.i1-IQ3_XS.gguf +3 -0
bella-bartender-heretic-3b.i1-IQ3_XXS.gguf +3 -0
bella-bartender-heretic-3b.i1-IQ4_NL.gguf +3 -0
bella-bartender-heretic-3b.i1-IQ4_XS.gguf +3 -0
bella-bartender-heretic-3b.i1-Q2_K.gguf +3 -0
bella-bartender-heretic-3b.i1-Q2_K_S.gguf +3 -0
bella-bartender-heretic-3b.i1-Q3_K_L.gguf +3 -0
bella-bartender-heretic-3b.i1-Q3_K_M.gguf +3 -0
bella-bartender-heretic-3b.i1-Q3_K_S.gguf +3 -0
bella-bartender-heretic-3b.i1-Q4_0.gguf +3 -0
bella-bartender-heretic-3b.i1-Q4_1.gguf +3 -0
bella-bartender-heretic-3b.i1-Q4_K_M.gguf +3 -0
bella-bartender-heretic-3b.i1-Q4_K_S.gguf +3 -0
bella-bartender-heretic-3b.i1-Q5_K_M.gguf +3 -0
bella-bartender-heretic-3b.i1-Q5_K_S.gguf +3 -0
bella-bartender-heretic-3b.i1-Q6_K.gguf +3 -0

.gitattributes CHANGED Viewed

@@ -34,3 +34,27 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 bella-bartender-heretic-3b.imatrix.gguf filter=lfs diff=lfs merge=lfs -text

 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 bella-bartender-heretic-3b.imatrix.gguf filter=lfs diff=lfs merge=lfs -text
+bella-bartender-heretic-3b.i1-IQ1_M.gguf filter=lfs diff=lfs merge=lfs -text
+bella-bartender-heretic-3b.i1-IQ1_S.gguf filter=lfs diff=lfs merge=lfs -text
+bella-bartender-heretic-3b.i1-IQ2_M.gguf filter=lfs diff=lfs merge=lfs -text
+bella-bartender-heretic-3b.i1-IQ2_S.gguf filter=lfs diff=lfs merge=lfs -text
+bella-bartender-heretic-3b.i1-IQ2_XS.gguf filter=lfs diff=lfs merge=lfs -text
+bella-bartender-heretic-3b.i1-IQ2_XXS.gguf filter=lfs diff=lfs merge=lfs -text
+bella-bartender-heretic-3b.i1-IQ3_M.gguf filter=lfs diff=lfs merge=lfs -text
+bella-bartender-heretic-3b.i1-IQ3_S.gguf filter=lfs diff=lfs merge=lfs -text
+bella-bartender-heretic-3b.i1-IQ3_XS.gguf filter=lfs diff=lfs merge=lfs -text
+bella-bartender-heretic-3b.i1-IQ3_XXS.gguf filter=lfs diff=lfs merge=lfs -text
+bella-bartender-heretic-3b.i1-IQ4_NL.gguf filter=lfs diff=lfs merge=lfs -text
+bella-bartender-heretic-3b.i1-IQ4_XS.gguf filter=lfs diff=lfs merge=lfs -text
+bella-bartender-heretic-3b.i1-Q2_K.gguf filter=lfs diff=lfs merge=lfs -text
+bella-bartender-heretic-3b.i1-Q2_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+bella-bartender-heretic-3b.i1-Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text
+bella-bartender-heretic-3b.i1-Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+bella-bartender-heretic-3b.i1-Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+bella-bartender-heretic-3b.i1-Q4_0.gguf filter=lfs diff=lfs merge=lfs -text
+bella-bartender-heretic-3b.i1-Q4_1.gguf filter=lfs diff=lfs merge=lfs -text
+bella-bartender-heretic-3b.i1-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+bella-bartender-heretic-3b.i1-Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+bella-bartender-heretic-3b.i1-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+bella-bartender-heretic-3b.i1-Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+bella-bartender-heretic-3b.i1-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text

bella-bartender-heretic-3b.i1-IQ1_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:957ed7ffa969e9e16e7bc2241647d55502c6d55805aebc69b73fe6998e401419
+size 924188448

bella-bartender-heretic-3b.i1-IQ1_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:f3fde3d0205d4a0057d792a4ae22d5d0abfe85bf9d1a49d19b4f418c7659cc67
+size 868155168

bella-bartender-heretic-3b.i1-IQ2_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:4b33e485b241f94a185429f20c415e562759885b0ee2aef5babb50f233bee860
+size 1229029152

bella-bartender-heretic-3b.i1-IQ2_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:130be04f490607ffc880ad0208c4670407aa11d89900f24ff862820434dd5592
+size 1154318112

bella-bartender-heretic-3b.i1-IQ2_XS.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:4d4e46aabba573338e51f7e7ff11bc7891a313563ef36852d7d033b0e5f2b36b
+size 1100545824

bella-bartender-heretic-3b.i1-IQ2_XXS.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:64a1c8ee1427ada5a6ea49ebec1f5fe2da972809f436ed5483ed4aeb78549e45
+size 1017577248

bella-bartender-heretic-3b.i1-IQ3_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:3788e416baaff57369f7f1ff67d124657bc02e9161f373e2ab1fb40587989291
+size 1599665952

bella-bartender-heretic-3b.i1-IQ3_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:a22eab56c672cc327cc63c1f502891a41ca89a23322f3bdab14912790290f81f
+size 1542846240

bella-bartender-heretic-3b.i1-IQ3_XS.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:aeef6a23b9c69b04520afb6cb8da8d8b51414670a035e8aef0a187d32d384a98
+size 1476785952

bella-bartender-heretic-3b.i1-IQ3_XXS.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:d27d29176c493c9b59ebafdc32d59918f38a6ccc9abd4da5fa35132c64486dba
+size 1348763424

bella-bartender-heretic-3b.i1-IQ4_NL.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:f29fe5018175e006eb6867576482c380a7035e2e3e6ee5873feb852db6b7646f
+size 1917187872

bella-bartender-heretic-3b.i1-IQ4_XS.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:8f3d86586760757a657973774133f27fd69632862e3c0de5989e702ac46d329b
+size 1829107488

bella-bartender-heretic-3b.i1-Q2_K.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:d3404f166e017a7c0fe3fc8bf69a759c04e410f937a8187bf56fe396e8e2c2f7
+size 1363932960

bella-bartender-heretic-3b.i1-Q2_K_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:360088e474e10df8abaf3156d861abfed342085cbaaf40d06bfbd47e57be93be
+size 1274279712

bella-bartender-heretic-3b.i1-Q3_K_L.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:2ea0baf789a1b45ceb00a0b9da1e14edb40e01ad7fd9569d5ab1debd11d8aeb8
+size 1815344928

bella-bartender-heretic-3b.i1-Q3_K_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:b713a715473d07078404fe50c44d4f6613ee84159a4750a1663e7f3e89b033af
+size 1687156512

bella-bartender-heretic-3b.i1-Q3_K_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:ccb82ec272a3e2d82f61983274a89775dcfe593311f58241e45174488b37319a
+size 1542846240

bella-bartender-heretic-3b.i1-Q4_0.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:f4f89439f5229d86c34205fcc9e8073d4872ff843dcb88d1d37de0d95924a195
+size 1921906464

bella-bartender-heretic-3b.i1-Q4_1.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:721c88ab13af8ab4ee4df74d8182864f3762ccc6a67fa02cbc527276ff12f55f
+size 2093348640

bella-bartender-heretic-3b.i1-Q4_K_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:a8b28789c9f062dc6596ee85fe61cf3f9a45930e1e6bc462cd9d316caa1fc1ad
+size 2019374880

bella-bartender-heretic-3b.i1-Q4_K_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:79ec0fcb83e45567eecb4924343f29cf4843611e63682d3dbf57cf4d00e7cac7
+size 1928197920

bella-bartender-heretic-3b.i1-Q5_K_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:bb53b5200e953e25b64051ef9e60890f307e47b71cca89439999385bc843a149
+size 2322151200

bella-bartender-heretic-3b.i1-Q5_K_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:0c01589122503e7548f98f5a576efb808dbd8f9b1ae634dfd8934d6f9c538ec5
+size 2269509408

bella-bartender-heretic-3b.i1-Q6_K.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:11a50b89d755c612610d5dc1c7e89a5e73d64d9fd35df98c6939ec60b717826c
+size 2643851040