Instructions to use mradermacher/neutts-nano-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use mradermacher/neutts-nano-GGUF with Transformers:
# Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("mradermacher/neutts-nano-GGUF", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use mradermacher/neutts-nano-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf mradermacher/neutts-nano-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf mradermacher/neutts-nano-GGUF:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf mradermacher/neutts-nano-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf mradermacher/neutts-nano-GGUF:Q4_K_M
Use pre-built binary
# Download pre-built binary from: # https://github.com/ggerganov/llama.cpp/releases # Start a local OpenAI-compatible server with a web UI: ./llama-server -hf mradermacher/neutts-nano-GGUF:Q4_K_M # Run inference directly in the terminal: ./llama-cli -hf mradermacher/neutts-nano-GGUF:Q4_K_M
Build from source code
git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-server llama-cli # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama-server -hf mradermacher/neutts-nano-GGUF:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama-cli -hf mradermacher/neutts-nano-GGUF:Q4_K_M
Use Docker
docker model run hf.co/mradermacher/neutts-nano-GGUF:Q4_K_M
- LM Studio
- Jan
- Ollama
How to use mradermacher/neutts-nano-GGUF with Ollama:
ollama run hf.co/mradermacher/neutts-nano-GGUF:Q4_K_M
- Unsloth Desktop
- Docker Model Runner
How to use mradermacher/neutts-nano-GGUF with Docker Model Runner:
docker model run hf.co/mradermacher/neutts-nano-GGUF:Q4_K_M
- Lemonade
How to use mradermacher/neutts-nano-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull mradermacher/neutts-nano-GGUF:Q4_K_M
Run and chat with the model
lemonade run user.neutts-nano-GGUF-Q4_K_M
List all available models
lemonade list
- Atomic Chat
uploaded from leia
Browse files- .gitattributes +12 -0
- neutts-nano.IQ4_XS.gguf +3 -0
- neutts-nano.Q2_K.gguf +3 -0
- neutts-nano.Q3_K_L.gguf +3 -0
- neutts-nano.Q3_K_M.gguf +3 -0
- neutts-nano.Q3_K_S.gguf +3 -0
- neutts-nano.Q4_K_M.gguf +3 -0
- neutts-nano.Q4_K_S.gguf +3 -0
- neutts-nano.Q5_K_M.gguf +3 -0
- neutts-nano.Q5_K_S.gguf +3 -0
- neutts-nano.Q6_K.gguf +3 -0
- neutts-nano.Q8_0.gguf +3 -0
- neutts-nano.f16.gguf +3 -0
.gitattributes
CHANGED
|
@@ -33,3 +33,15 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
+
neutts-nano.IQ4_XS.gguf filter=lfs diff=lfs merge=lfs -text
|
| 37 |
+
neutts-nano.Q2_K.gguf filter=lfs diff=lfs merge=lfs -text
|
| 38 |
+
neutts-nano.Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text
|
| 39 |
+
neutts-nano.Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
|
| 40 |
+
neutts-nano.Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text
|
| 41 |
+
neutts-nano.Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
|
| 42 |
+
neutts-nano.Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
|
| 43 |
+
neutts-nano.Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
|
| 44 |
+
neutts-nano.Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text
|
| 45 |
+
neutts-nano.Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
|
| 46 |
+
neutts-nano.Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
|
| 47 |
+
neutts-nano.f16.gguf filter=lfs diff=lfs merge=lfs -text
|
neutts-nano.IQ4_XS.gguf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:6f9133a321c3ae62c8dfe95d00cfc302a235733d2db100925f1d76babadee14a
|
| 3 |
+
size 256917664
|
neutts-nano.Q2_K.gguf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:642b9ed18767b852a1ca88b6bb49796974e1b02f5e1b2a49ba57ba17d0d71d34
|
| 3 |
+
size 253060768
|
neutts-nano.Q3_K_L.gguf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:692290049f58a0ad37f9baff0224c035e8776613e8afce2df7c6c49c054bd11d
|
| 3 |
+
size 263262880
|
neutts-nano.Q3_K_M.gguf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:344dc553e63e1e7351647eb14eb9227b1c42f2f0840671fcbe5d91ef750f7c37
|
| 3 |
+
size 258797728
|
neutts-nano.Q3_K_S.gguf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:3e082fbe5a8e4dac2ad98fb7091c14c85055698cddfc2308759b132aae35b3df
|
| 3 |
+
size 253060768
|
neutts-nano.Q4_K_M.gguf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:ae04f938246fdc0a12546ba069f2ef62a55d912e5018dafe08c3152da5c705f6
|
| 3 |
+
size 286497568
|
neutts-nano.Q4_K_S.gguf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:862a49c7bdc93d8ed79b12886add2fa92dd3e4e8de751ea26f819d82f8ff19b6
|
| 3 |
+
size 282419488
|
neutts-nano.Q5_K_M.gguf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:0da76a25f2206272f50671c45283a4c3017666ae2f5240b29c6cc87ad44ac4f7
|
| 3 |
+
size 300706912
|
neutts-nano.Q5_K_S.gguf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:586492add6a22ba1cf78f16c2a05ef73654b7161482132477dc06f11fb776872
|
| 3 |
+
size 298177120
|
neutts-nano.Q6_K.gguf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:cc908562d9494b44c890a5a0a586d93cae6e91ef14a6f714d529c7426cf21f91
|
| 3 |
+
size 363915424
|
neutts-nano.Q8_0.gguf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:634100e8830315d130f70f4f0c3d7527677b4082d875e31066560af2b34a6cd4
|
| 3 |
+
size 371629216
|
neutts-nano.f16.gguf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:9bf66c60e27cced2f17d3d3d3403bcbd522e908450d187c346456e32895b741f
|
| 3 |
+
size 690911776
|