Instructions to use tdrussell/Mixtral-8x22B-Capyboros-v1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use tdrussell/Mixtral-8x22B-Capyboros-v1 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="tdrussell/Mixtral-8x22B-Capyboros-v1")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("tdrussell/Mixtral-8x22B-Capyboros-v1")
model = AutoModelForCausalLM.from_pretrained("tdrussell/Mixtral-8x22B-Capyboros-v1", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use tdrussell/Mixtral-8x22B-Capyboros-v1 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "tdrussell/Mixtral-8x22B-Capyboros-v1"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tdrussell/Mixtral-8x22B-Capyboros-v1",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/tdrussell/Mixtral-8x22B-Capyboros-v1

SGLang

How to use tdrussell/Mixtral-8x22B-Capyboros-v1 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "tdrussell/Mixtral-8x22B-Capyboros-v1" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tdrussell/Mixtral-8x22B-Capyboros-v1",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "tdrussell/Mixtral-8x22B-Capyboros-v1" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tdrussell/Mixtral-8x22B-Capyboros-v1",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use tdrussell/Mixtral-8x22B-Capyboros-v1 with Docker Model Runner:
```
docker model run hf.co/tdrussell/Mixtral-8x22B-Capyboros-v1
```

tdrussell commited on Apr 15, 2024

Commit

77a9dba

verified ·

1 Parent(s): 9c22328

Upload 20 files

Browse files

Files changed (20) hide show

model-00021-of-00059.safetensors +3 -0
model-00022-of-00059.safetensors +3 -0
model-00023-of-00059.safetensors +3 -0
model-00024-of-00059.safetensors +3 -0
model-00025-of-00059.safetensors +3 -0
model-00026-of-00059.safetensors +3 -0
model-00027-of-00059.safetensors +3 -0
model-00028-of-00059.safetensors +3 -0
model-00029-of-00059.safetensors +3 -0
model-00030-of-00059.safetensors +3 -0
model-00031-of-00059.safetensors +3 -0
model-00032-of-00059.safetensors +3 -0
model-00033-of-00059.safetensors +3 -0
model-00034-of-00059.safetensors +3 -0
model-00035-of-00059.safetensors +3 -0
model-00036-of-00059.safetensors +3 -0
model-00037-of-00059.safetensors +3 -0
model-00038-of-00059.safetensors +3 -0
model-00039-of-00059.safetensors +3 -0
model-00040-of-00059.safetensors +3 -0

model-00021-of-00059.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:8e2da184d27465732e359fd294de92b4bda704d8662ab0f5698e3b8fc2adf0ff
+size 4806799152

model-00022-of-00059.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:d95bf9fe9b307e6eee26ac83faa0313b2dd4f739530150d26c00b95284f87c3f
+size 4806799152

model-00023-of-00059.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:670b30192bcfec01922b30dea66a3fa22fbc46f14395bc8b6e042707be99bbf8
+size 4806799152

model-00024-of-00059.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:a154ce65bcdaee6d12cfe37059672c51283c885e6499fe3ce6e4f41f57ba1686
+size 4932529864

model-00025-of-00059.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:64caa09441e13e8fe5d79c82255047e7e996acd0edda2ff1e8230221e1d1b61d
+size 4995542848

model-00026-of-00059.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:e34f5ec8149c80cc426808db6787489542d93b9b7f59ae369510e75fcefba027
+size 4995542848

model-00027-of-00059.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:f50d16a62de92aace1ab882de84af53ffd797419136afef34ab10f2758ef208c
+size 4932628288

model-00028-of-00059.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:9a362f1d59340f7904f73a3817e5b124973e8ac3d3084df0757072d23faf3bd7
+size 4806774344

model-00029-of-00059.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:42b43875c8a5d68cb9ec8bb3eb2a5cf5b2574fb7b758c8ceca54f104a040bef0
+size 4806799144

model-00030-of-00059.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:ace982b70805128722feb7976e0a6f935aa8a64e8a7ef6bcd823f035f9751e32
+size 4806799144

model-00031-of-00059.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:dd64dd4e729f7366367409127a10881c2d903fc20f6c8b73760a56984eed032a
+size 4806799144

model-00032-of-00059.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:65c37dcdd0aa7b14a1daab94f58498075d5978223c2c29299ab9edabd226294b
+size 4806799144

model-00033-of-00059.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:1d3e9cac10299c3ce445d4e29ed8cf9f0ee0c288f961a88e7770da01207bc829
+size 4806799152

model-00034-of-00059.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:439e6d158545b3fd8d4bb77f959dae10946a425a6c6024f41c27c2deec1ad8dc
+size 4806799152

model-00035-of-00059.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:b2d702a8a1371a90b69568fca1045eaa584a3f695654fe53d1e4e4eac92abf8e
+size 4806799152

model-00036-of-00059.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:3afbfde8d28d0d53c96da7f838f0fa8b586e26f5a07b647a79f7750b94b46525
+size 4806799152

model-00037-of-00059.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:ee3600d900d92a6fdb8ff36335d9129c633ed01b479e641d20c83824f39bcd0b
+size 4806799152

model-00038-of-00059.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:940a747079ec19fafcc6ae0684028a3ddedfc473f6154dffe9ee34e3319b570d
+size 4806799152

model-00039-of-00059.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:e849876a7414989c5c5aab83773aed9da592d6a63fe4a1fd07b8e73a686b81cd
+size 4806799152

model-00040-of-00059.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:a6d9ea445f8174e7b341e3d8bbcfcffe21cdf3b600d719bc44f6594447c96c92
+size 4806799152