Instructions to use sophosympatheia/Midnight-Miqu-70B-v1.5 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use sophosympatheia/Midnight-Miqu-70B-v1.5 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="sophosympatheia/Midnight-Miqu-70B-v1.5")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("sophosympatheia/Midnight-Miqu-70B-v1.5")
model = AutoModelForCausalLM.from_pretrained("sophosympatheia/Midnight-Miqu-70B-v1.5", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use sophosympatheia/Midnight-Miqu-70B-v1.5 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "sophosympatheia/Midnight-Miqu-70B-v1.5"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "sophosympatheia/Midnight-Miqu-70B-v1.5",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/sophosympatheia/Midnight-Miqu-70B-v1.5

SGLang

How to use sophosympatheia/Midnight-Miqu-70B-v1.5 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "sophosympatheia/Midnight-Miqu-70B-v1.5" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "sophosympatheia/Midnight-Miqu-70B-v1.5",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "sophosympatheia/Midnight-Miqu-70B-v1.5" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "sophosympatheia/Midnight-Miqu-70B-v1.5",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use sophosympatheia/Midnight-Miqu-70B-v1.5 with Docker Model Runner:
```
docker model run hf.co/sophosympatheia/Midnight-Miqu-70B-v1.5
```

Midnight-Miqu-70B-v1.5 / README.md

Commit History

Update README.md

7488897
verified

sophosympatheia commited on Dec 10, 2024

Adding Evaluation Results (#12)

90d9339
verified

sophosympatheia

leaderboard-pr-bot commited on Oct 30, 2024

Update README.md

f6062ca
verified

sophosympatheia commited on Apr 16, 2024

Fix typo (#7)

658d919
verified

sophosympatheia

nonetrix commited on Apr 13, 2024

Metadata: encode merged models as `base_model` (#4)

f106f7e
verified

sophosympatheia

julien-c HF Staff commited on Mar 23, 2024

Update README.md

444fc91
verified

sophosympatheia commited on Mar 15, 2024

Update README.md

652128b
verified

sophosympatheia commited on Mar 15, 2024

Update README.md

3777ed1
verified

sophosympatheia commited on Mar 15, 2024

Update README.md

9726bce
verified

sophosympatheia commited on Mar 14, 2024

Update README.md

090f9f4
verified

sophosympatheia commited on Mar 14, 2024

Upload model weights (#1)

da9b25a
verified

sophosympatheia commited on Mar 12, 2024

initial commit

8e9e467
verified

sophosympatheia commited on Mar 11, 2024

Commit History

Update README.md 7488897 verified

Adding Evaluation Results (#12) 90d9339 verified

Update README.md f6062ca verified

Fix typo (#7) 658d919 verified

Metadata: encode merged models as `base_model` (#4) f106f7e verified

Update README.md 444fc91 verified

Update README.md 652128b verified

Update README.md 3777ed1 verified

Update README.md 9726bce verified

Update README.md 090f9f4 verified

Upload model weights (#1) da9b25a verified

initial commit 8e9e467 verified

Update README.md

7488897
verified

Adding Evaluation Results (#12)

90d9339
verified

Update README.md

f6062ca
verified

Fix typo (#7)

658d919
verified

Metadata: encode merged models as `base_model` (#4)

f106f7e
verified

Update README.md

444fc91
verified

Update README.md

652128b
verified

Update README.md

3777ed1
verified

Update README.md

9726bce
verified

Update README.md

090f9f4
verified

Upload model weights (#1)

da9b25a
verified

initial commit

8e9e467
verified