Instructions to use gaianet/Yi-1.5-34B-Chat-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use gaianet/Yi-1.5-34B-Chat-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf gaianet/Yi-1.5-34B-Chat-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf gaianet/Yi-1.5-34B-Chat-GGUF:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf gaianet/Yi-1.5-34B-Chat-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf gaianet/Yi-1.5-34B-Chat-GGUF:Q4_K_M
Use pre-built binary
# Download pre-built binary from: # https://github.com/ggerganov/llama.cpp/releases # Start a local OpenAI-compatible server with a web UI: ./llama-server -hf gaianet/Yi-1.5-34B-Chat-GGUF:Q4_K_M # Run inference directly in the terminal: ./llama-cli -hf gaianet/Yi-1.5-34B-Chat-GGUF:Q4_K_M
Build from source code
git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-server llama-cli # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama-server -hf gaianet/Yi-1.5-34B-Chat-GGUF:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama-cli -hf gaianet/Yi-1.5-34B-Chat-GGUF:Q4_K_M
Use Docker
docker model run hf.co/gaianet/Yi-1.5-34B-Chat-GGUF:Q4_K_M
- LM Studio
- Jan
- vLLM
How to use gaianet/Yi-1.5-34B-Chat-GGUF with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "gaianet/Yi-1.5-34B-Chat-GGUF" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "gaianet/Yi-1.5-34B-Chat-GGUF", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/gaianet/Yi-1.5-34B-Chat-GGUF:Q4_K_M
- Ollama
How to use gaianet/Yi-1.5-34B-Chat-GGUF with Ollama:
ollama run hf.co/gaianet/Yi-1.5-34B-Chat-GGUF:Q4_K_M
- Unsloth Desktop
- Docker Model Runner
How to use gaianet/Yi-1.5-34B-Chat-GGUF with Docker Model Runner:
docker model run hf.co/gaianet/Yi-1.5-34B-Chat-GGUF:Q4_K_M
- Lemonade
How to use gaianet/Yi-1.5-34B-Chat-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull gaianet/Yi-1.5-34B-Chat-GGUF:Q4_K_M
Run and chat with the model
lemonade run user.Yi-1.5-34B-Chat-GGUF-Q4_K_M
List all available models
lemonade list
- Atomic Chat
Upload models
Browse files- .gitattributes +15 -0
- Yi-1.5-34B-Chat-Q2_K.gguf +3 -0
- Yi-1.5-34B-Chat-Q3_K_L.gguf +3 -0
- Yi-1.5-34B-Chat-Q3_K_M.gguf +3 -0
- Yi-1.5-34B-Chat-Q3_K_S.gguf +3 -0
- Yi-1.5-34B-Chat-Q4_0.gguf +3 -0
- Yi-1.5-34B-Chat-Q4_K_M.gguf +3 -0
- Yi-1.5-34B-Chat-Q4_K_S.gguf +3 -0
- Yi-1.5-34B-Chat-Q5_0.gguf +3 -0
- Yi-1.5-34B-Chat-Q5_K_M.gguf +3 -0
- Yi-1.5-34B-Chat-Q5_K_S.gguf +3 -0
- Yi-1.5-34B-Chat-Q6_K.gguf +3 -0
- Yi-1.5-34B-Chat-Q8_0.gguf +3 -0
- Yi-1.5-34B-Chat-f16-00001-of-00003.gguf +3 -0
- Yi-1.5-34B-Chat-f16-00002-of-00003.gguf +3 -0
- Yi-1.5-34B-Chat-f16-00003-of-00003.gguf +3 -0
.gitattributes
CHANGED
|
@@ -33,3 +33,18 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
+
Yi-1.5-34B-Chat-Q2_K.gguf filter=lfs diff=lfs merge=lfs -text
|
| 37 |
+
Yi-1.5-34B-Chat-Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text
|
| 38 |
+
Yi-1.5-34B-Chat-Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
|
| 39 |
+
Yi-1.5-34B-Chat-Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text
|
| 40 |
+
Yi-1.5-34B-Chat-Q4_0.gguf filter=lfs diff=lfs merge=lfs -text
|
| 41 |
+
Yi-1.5-34B-Chat-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
|
| 42 |
+
Yi-1.5-34B-Chat-Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
|
| 43 |
+
Yi-1.5-34B-Chat-Q5_0.gguf filter=lfs diff=lfs merge=lfs -text
|
| 44 |
+
Yi-1.5-34B-Chat-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
|
| 45 |
+
Yi-1.5-34B-Chat-Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text
|
| 46 |
+
Yi-1.5-34B-Chat-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
|
| 47 |
+
Yi-1.5-34B-Chat-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
|
| 48 |
+
Yi-1.5-34B-Chat-f16-00001-of-00003.gguf filter=lfs diff=lfs merge=lfs -text
|
| 49 |
+
Yi-1.5-34B-Chat-f16-00002-of-00003.gguf filter=lfs diff=lfs merge=lfs -text
|
| 50 |
+
Yi-1.5-34B-Chat-f16-00003-of-00003.gguf filter=lfs diff=lfs merge=lfs -text
|
Yi-1.5-34B-Chat-Q2_K.gguf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:1cb8915f13f2c18fbde911d957b74eefcc64c0d8656d57d7d026e9d017f4e402
|
| 3 |
+
size 12825233984
|
Yi-1.5-34B-Chat-Q3_K_L.gguf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:be1d60202ed11288169d9af07a216c9c1e373a81d9177c68cfb46d66eb05440e
|
| 3 |
+
size 18139445824
|
Yi-1.5-34B-Chat-Q3_K_M.gguf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:d5e5cc82e458855c406900a6558f97ad45bb3365ba623e7b63ff943d80bcfe80
|
| 3 |
+
size 16654924352
|
Yi-1.5-34B-Chat-Q3_K_S.gguf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:1545727ad369879e5c20715020b75c83dacf77e765bdf48dd874e1c668912679
|
| 3 |
+
size 14960294464
|
Yi-1.5-34B-Chat-Q4_0.gguf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:97d448dd39bfcd0324cc6ca3aca2fa2a15000323c470fa8a0b63c46d57784641
|
| 3 |
+
size 19466529344
|
Yi-1.5-34B-Chat-Q4_K_M.gguf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:4663581760f66b14fa4ca3427e29fcbb41845e1d070ca81b72da984a9995d564
|
| 3 |
+
size 20658711104
|
Yi-1.5-34B-Chat-Q4_K_S.gguf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:6c317f398f7da8aa8d4582a39c0c2d3f875082a16f4634aafbc8f25f0b10c13d
|
| 3 |
+
size 19598649920
|
Yi-1.5-34B-Chat-Q5_0.gguf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:2326d5a9db951af79c0528c06efa302a053d05eed354688d91d5fe54161d1150
|
| 3 |
+
size 23707691584
|
Yi-1.5-34B-Chat-Q5_K_M.gguf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:24e0d52785d7fbb6269791934cf9f22d79e9d3fcef223fc9c70e2266010754f7
|
| 3 |
+
size 24321845824
|
Yi-1.5-34B-Chat-Q5_K_S.gguf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:b59234f2807ddf61c1c8748d2209edb078f4369d3d1edef91777511c8549d5b3
|
| 3 |
+
size 23707691584
|
Yi-1.5-34B-Chat-Q6_K.gguf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:f231020b32f84aacbc0e08d6de14f2bdfc67a2d0c1aa6b5300a3ba12b0a6f9fe
|
| 3 |
+
size 28213926464
|
Yi-1.5-34B-Chat-Q8_0.gguf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:24fe8f9e46b6130915605ece8078e7609a4c24c9474bf477ce5ec1debda0c649
|
| 3 |
+
size 36542282304
|
Yi-1.5-34B-Chat-f16-00001-of-00003.gguf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:a31e4ef5d63acaca287fc401e02ea603e0623cf35502c9124e20853d85fca1e9
|
| 3 |
+
size 32159807232
|
Yi-1.5-34B-Chat-f16-00002-of-00003.gguf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:3c80f04a7968bc132d459c3fb3515f843d1cfc049355db438839cfbaf96d17c8
|
| 3 |
+
size 32136309888
|
Yi-1.5-34B-Chat-f16-00003-of-00003.gguf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:6a194d9b63cd024e07101cec24e430fd400f850b253a991f97d7de58becafd1d
|
| 3 |
+
size 4484962272
|