Instructions to use OrionStarAI/Orion-14B-Chat with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use OrionStarAI/Orion-14B-Chat with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="OrionStarAI/Orion-14B-Chat", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("OrionStarAI/Orion-14B-Chat", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use OrionStarAI/Orion-14B-Chat with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf OrionStarAI/Orion-14B-Chat
# Run inference directly in the terminal:
llama cli -hf OrionStarAI/Orion-14B-Chat

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf OrionStarAI/Orion-14B-Chat
# Run inference directly in the terminal:
llama cli -hf OrionStarAI/Orion-14B-Chat

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf OrionStarAI/Orion-14B-Chat
# Run inference directly in the terminal:
./llama-cli -hf OrionStarAI/Orion-14B-Chat

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf OrionStarAI/Orion-14B-Chat
# Run inference directly in the terminal:
./build/bin/llama-cli -hf OrionStarAI/Orion-14B-Chat

Use Docker

docker model run hf.co/OrionStarAI/Orion-14B-Chat

LM Studio
Jan

vLLM

How to use OrionStarAI/Orion-14B-Chat with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "OrionStarAI/Orion-14B-Chat"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "OrionStarAI/Orion-14B-Chat",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/OrionStarAI/Orion-14B-Chat

SGLang

How to use OrionStarAI/Orion-14B-Chat with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "OrionStarAI/Orion-14B-Chat" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "OrionStarAI/Orion-14B-Chat",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "OrionStarAI/Orion-14B-Chat" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "OrionStarAI/Orion-14B-Chat",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Ollama
How to use OrionStarAI/Orion-14B-Chat with Ollama:
```
ollama run hf.co/OrionStarAI/Orion-14B-Chat
```
Unsloth Desktop
Docker Model Runner
How to use OrionStarAI/Orion-14B-Chat with Docker Model Runner:
```
docker model run hf.co/OrionStarAI/Orion-14B-Chat
```

Lemonade

How to use OrionStarAI/Orion-14B-Chat with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull OrionStarAI/Orion-14B-Chat

Run and chat with the model

lemonade run user.Orion-14B-Chat-{{QUANT_TAG}}

List all available models

lemonade list

Atomic Chat

renillhuang commited on Jan 25, 2024

Commit

210843f

verified ·

1 Parent(s): 8f474a2

Update README_zh.md

Browse files

Files changed (1) hide show

README_zh.md +32 -4

README_zh.md CHANGED Viewed

@@ -32,7 +32,7 @@
 - [📖 模型介绍](#zh_model-introduction)
 - [🔗 下载路径](#zh_model-download)
 - [🔖 评估结果](#zh_model-benchmark)
-- [📊 模型推理](#zh_model-inference)
 - [📜 声明协议](#zh_declarations-license)
 - [🥇 企业介绍](#zh_company-introduction)
@@ -266,10 +266,38 @@ CUDA_VISIBLE_DEVICES=0 python demo/text_generation_base.py --model OrionStarAI/O
 CUDA_VISIBLE_DEVICES=0 python demo/text_generation.py --model OrionStarAI/Orion-14B-Chat --tokenizer OrionStarAI/Orion-14B-Chat --prompt 你好,你叫什么名字
 ```
-## 4.4. 示例输出
-### 4.4.1. 闲聊
 `````
 用户：你好,你叫什么名字
@@ -296,7 +324,7 @@ Orion-14B：从前有一个小男孩叫杰克,他住在一个小村庄里。杰
   这个故事告诉我们,只要有勇气和决心,我们就能克服一切困难,实现自己的梦想。
 `````
-### 4.4.2. 日韩
 `````
 用户：自己を紹介してください

 - [📖 模型介绍](#zh_model-introduction)
 - [🔗 下载路径](#zh_model-download)
 - [🔖 评估结果](#zh_model-benchmark)
+- [📊 模型推理](#zh_model-inference)[<img src="./assets/imgs/vllm.png" alt="vllm" height="20"/>](#vllm) [<img src="./assets/imgs/llama_cpp.png" alt="llamacpp" height="20"/>](#llama-cpp)
 - [📜 声明协议](#zh_declarations-license)
 - [🥇 企业介绍](#zh_company-introduction)
 CUDA_VISIBLE_DEVICES=0 python demo/text_generation.py --model OrionStarAI/Orion-14B-Chat --tokenizer OrionStarAI/Orion-14B-Chat --prompt 你好,你叫什么名字
 ```
+## 4.4. 使用vllm推理
+- 工程地址<br>
+  https://github.com/vllm-project/vllm
+- 拉取请求<br>
+  https://github.com/vllm-project/vllm/pull/2539
+<a name="llama-cpp"></a><br>
+## 4.5. 使用llama.cpp推理
+- 工程地址<br>
+  https://github.com/ggerganov/llama.cpp
+- 拉取请求<br>
+  https://github.com/ggerganov/llama.cpp/pull/5118
+- 如何转换GGUF格式模型
+  ```shell
+  python convert-hf-to-gguf.py path/to/Orion-14B-Chat --outfile chat.gguf
+  ```
+- 如何运行GGUF格式模型推理
+  ```shell
+  ./main --frequency-penalty 0.5 --frequency-penalty 0.5 --top-k 5 --top-p 0.9 -m chat.gguf -p "Building a website can be done in 10 simple steps:\nStep 1:" -n 400 -e
+  ```
+## 4.6. 示例输出
+### 4.6.1. 闲聊
 `````
 用户：你好,你叫什么名字
   这个故事告诉我们,只要有勇气和决心,我们就能克服一切困难,实现自己的梦想。
 `````
+### 4.6.2. 日韩
 `````
 用户：自己を紹介してください