Instructions to use PracticeLLM/SOLAR-tail-10.7B-Merge-v1.0 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use PracticeLLM/SOLAR-tail-10.7B-Merge-v1.0 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="PracticeLLM/SOLAR-tail-10.7B-Merge-v1.0")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("PracticeLLM/SOLAR-tail-10.7B-Merge-v1.0")
model = AutoModelForCausalLM.from_pretrained("PracticeLLM/SOLAR-tail-10.7B-Merge-v1.0", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use PracticeLLM/SOLAR-tail-10.7B-Merge-v1.0 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "PracticeLLM/SOLAR-tail-10.7B-Merge-v1.0"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "PracticeLLM/SOLAR-tail-10.7B-Merge-v1.0",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/PracticeLLM/SOLAR-tail-10.7B-Merge-v1.0

SGLang

How to use PracticeLLM/SOLAR-tail-10.7B-Merge-v1.0 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "PracticeLLM/SOLAR-tail-10.7B-Merge-v1.0" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "PracticeLLM/SOLAR-tail-10.7B-Merge-v1.0",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "PracticeLLM/SOLAR-tail-10.7B-Merge-v1.0" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "PracticeLLM/SOLAR-tail-10.7B-Merge-v1.0",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use PracticeLLM/SOLAR-tail-10.7B-Merge-v1.0 with Docker Model Runner:
```
docker model run hf.co/PracticeLLM/SOLAR-tail-10.7B-Merge-v1.0
```

kyujinpy commited on Dec 27, 2023

Commit

ff15ee2

1 Parent(s): 714a6cc

Upload README.md

Browse files

Files changed (1) hide show

README.md +49 -4

README.md CHANGED Viewed

@@ -10,15 +10,41 @@ license: cc-by-nc-sa-4.0
 ## Model Details
-**Model Developers** PracticeLLM (kyujinpy)
 **Method**
-(will update. wait!)
 # **Model Benchmark**
-## Open leaderboard
-- Follow up as [link](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard).
 | Model | Average | ARC | HellaSwag | MMLU | TruthfulQA | Ko-CommonGenV2 |
 | --- | --- | --- | --- | --- | --- | --- |
@@ -27,7 +53,26 @@ license: cc-by-nc-sa-4.0
 | beomi/Yi-Ko-6B | 48.79 | 41.04 | 53.39 | 46.28 | 41.64 | 61.63 |
 | mistralai/Mistral-7B-v0.1 | 46.89 | 38.14 | 48.19 | 45.20 | 46.13 | 56.79 |
 # Implementation Code
 ```python
 ### KO-Platypus

 ## Model Details
+**Model Developers** Kyujin Han (kyujinpy)
 **Method**
+Using [Mergekit](https://github.com/cg123/mergekit).
+- [upstage/SOLAR-10.7B-v1.0](https://huggingface.co/upstage/SOLAR-10.7B-v1.0)
+- [Yhyu13/LMCocktail-10.7B-v1](Yhyu13/LMCocktail-10.7B-v1)
+**Merge config**
+```
+slices:
+  - sources:
+      - model: upstage/SOLAR-10.7B-v1.0
+        layer_range: [0, 48]
+      - model: Yhyu13/LMCocktail-10.7B-v1
+        layer_range: [0, 48]
+merge_method: slerp
+base_model: upstage/SOLAR-10.7B-v1.0
+parameters:
+  t:
+    - filter: self_attn
+      value: [0, 0.5, 0.3, 0.7, 1]
+    - filter: mlp
+      value: [1, 0.5, 0.7, 0.3, 0]
+    - value: 0.5 # fallback for rest of tensors
+tokenizer_source: union
+dtype: float16
+```
 # **Model Benchmark**
+## Open Ko leaderboard
+- Follow up as [Ko-link](https://huggingface.co/spaces/upstage/open-ko-llm-leaderboard).
 | Model | Average | ARC | HellaSwag | MMLU | TruthfulQA | Ko-CommonGenV2 |
 | --- | --- | --- | --- | --- | --- | --- |
 | beomi/Yi-Ko-6B | 48.79 | 41.04 | 53.39 | 46.28 | 41.64 | 61.63 |
 | mistralai/Mistral-7B-v0.1 | 46.89 | 38.14 | 48.19 | 45.20 | 46.13 | 56.79 |
+- Follow up as [En-link](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard).
+## lm-evaluation-harness
+```
+gpt2 (pretrained=PracticeLLM/SOLAR-tail-10.7B-Merge-v1.0), limit: None, provide_description: False, num_fewshot: 0, batch_size: None
+|      Task      |Version| Metric |Value |   |Stderr|
+|----------------|------:|--------|-----:|---|-----:|
+|kobest_boolq    |      0|acc     |0.5021|±  |0.0133|
+|                |       |macro_f1|0.3343|±  |0.0059|
+|kobest_copa     |      0|acc     |0.6220|±  |0.0153|
+|                |       |macro_f1|0.6217|±  |0.0154|
+|kobest_hellaswag|      0|acc     |0.4380|±  |0.0222|
+|                |       |acc_norm|0.5380|±  |0.0223|
+|                |       |macro_f1|0.4366|±  |0.0222|
+|kobest_sentineg |      0|acc     |0.4962|±  |0.0251|
+|                |       |macro_f1|0.3316|±  |0.0113|
+```
 # Implementation Code
 ```python
 ### KO-Platypus