Instructions to use mradermacher/Llama-3.2-3B-optimized-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use mradermacher/Llama-3.2-3B-optimized-GGUF with Transformers:

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("mradermacher/Llama-3.2-3B-optimized-GGUF", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use mradermacher/Llama-3.2-3B-optimized-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf mradermacher/Llama-3.2-3B-optimized-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf mradermacher/Llama-3.2-3B-optimized-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf mradermacher/Llama-3.2-3B-optimized-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf mradermacher/Llama-3.2-3B-optimized-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf mradermacher/Llama-3.2-3B-optimized-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf mradermacher/Llama-3.2-3B-optimized-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf mradermacher/Llama-3.2-3B-optimized-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf mradermacher/Llama-3.2-3B-optimized-GGUF:Q4_K_M

Use Docker

docker model run hf.co/mradermacher/Llama-3.2-3B-optimized-GGUF:Q4_K_M

LM Studio
Jan
Ollama
How to use mradermacher/Llama-3.2-3B-optimized-GGUF with Ollama:
```
ollama run hf.co/mradermacher/Llama-3.2-3B-optimized-GGUF:Q4_K_M
```
Unsloth Desktop
Docker Model Runner
How to use mradermacher/Llama-3.2-3B-optimized-GGUF with Docker Model Runner:
```
docker model run hf.co/mradermacher/Llama-3.2-3B-optimized-GGUF:Q4_K_M
```

Lemonade

How to use mradermacher/Llama-3.2-3B-optimized-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull mradermacher/Llama-3.2-3B-optimized-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Llama-3.2-3B-optimized-GGUF-Q4_K_M

List all available models

lemonade list

Atomic Chat

mradermacher commited on Apr 26

Commit

b354916

verified ·

1 Parent(s): a63fa5e

uploaded from marco

Browse files

Files changed (13) hide show

.gitattributes +12 -0
Llama-3.2-3B-optimized.IQ4_XS.gguf +3 -0
Llama-3.2-3B-optimized.Q2_K.gguf +3 -0
Llama-3.2-3B-optimized.Q3_K_L.gguf +3 -0
Llama-3.2-3B-optimized.Q3_K_M.gguf +3 -0
Llama-3.2-3B-optimized.Q3_K_S.gguf +3 -0
Llama-3.2-3B-optimized.Q4_K_M.gguf +3 -0
Llama-3.2-3B-optimized.Q4_K_S.gguf +3 -0
Llama-3.2-3B-optimized.Q5_K_M.gguf +3 -0
Llama-3.2-3B-optimized.Q5_K_S.gguf +3 -0
Llama-3.2-3B-optimized.Q6_K.gguf +3 -0
Llama-3.2-3B-optimized.Q8_0.gguf +3 -0
Llama-3.2-3B-optimized.f16.gguf +3 -0

.gitattributes CHANGED Viewed

@@ -33,3 +33,15 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text

 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+Llama-3.2-3B-optimized.IQ4_XS.gguf filter=lfs diff=lfs merge=lfs -text
+Llama-3.2-3B-optimized.Q2_K.gguf filter=lfs diff=lfs merge=lfs -text
+Llama-3.2-3B-optimized.Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text
+Llama-3.2-3B-optimized.Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+Llama-3.2-3B-optimized.Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+Llama-3.2-3B-optimized.Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+Llama-3.2-3B-optimized.Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+Llama-3.2-3B-optimized.Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+Llama-3.2-3B-optimized.Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+Llama-3.2-3B-optimized.Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
+Llama-3.2-3B-optimized.Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
+Llama-3.2-3B-optimized.f16.gguf filter=lfs diff=lfs merge=lfs -text

Llama-3.2-3B-optimized.IQ4_XS.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:8027a02c96b1e929a0b056d6d7d2fabbd3cf933774a4301741a361a6c3d25eb9
+size 1836338176

Llama-3.2-3B-optimized.Q2_K.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:f73775fe675a5f437bda2df4f42d39f16ebaaf6f404b1fb54ea5b9c85e47f3d0
+size 1359367168

Llama-3.2-3B-optimized.Q3_K_L.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:f7233f8fce5a3a05c6cbd0d525a1c99cc20ae37e8c314a1a1ede0dda582507e8
+size 1810779136

Llama-3.2-3B-optimized.Q3_K_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:f8e2255343ba9e4cbb05c3aa95951aba95e87d3468aa189ebe821eef024f5b7b
+size 1682590720

Llama-3.2-3B-optimized.Q3_K_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:afd638bd8b1755cf51c02295591f12a12b055dd367d9aa51dc733aea590bfafb
+size 1538280448

Llama-3.2-3B-optimized.Q4_K_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:33cd39bbee5e21f2c41d26ae31d4aefbef8f257abc24b3700698659a38c569a6
+size 2014809088

Llama-3.2-3B-optimized.Q4_K_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:a3a23864d0a41422faabd59f15c4f8f950b909e0517d3ee03ac5614844ae48c3
+size 1923632128

Llama-3.2-3B-optimized.Q5_K_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:8f0ab23ddffd5df5296455e3adbe300d80a174eb08d482a990849755b374d9ff
+size 2317585408

Llama-3.2-3B-optimized.Q5_K_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:06e3662a102e5ed6bb9ea0cb51f7eaa98f8f1227c5398237b49c112dc7b92aae
+size 2264943616

Llama-3.2-3B-optimized.Q6_K.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:d8b9f523439b0f005c3f7abeed04a5b8c360c292b28161ccbfbba75c8ad3c24a
+size 2639285248

Llama-3.2-3B-optimized.Q8_0.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:7b3166b08ca9f58266a8fdc66d790f632f3a05931490afb8b8510c44c3703c68
+size 3417330688

Llama-3.2-3B-optimized.f16.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:a2ff3468f697d2aa4f55cf4391ae20102f020a8b8b6399d1cff767684c3d9f2d
+size 6429119488