Instructions to use yentinglin/Taiwan-LLaMa-v1.0 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use yentinglin/Taiwan-LLaMa-v1.0 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="yentinglin/Taiwan-LLaMa-v1.0")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("yentinglin/Taiwan-LLaMa-v1.0")
model = AutoModelForCausalLM.from_pretrained("yentinglin/Taiwan-LLaMa-v1.0", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use yentinglin/Taiwan-LLaMa-v1.0 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "yentinglin/Taiwan-LLaMa-v1.0"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "yentinglin/Taiwan-LLaMa-v1.0",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/yentinglin/Taiwan-LLaMa-v1.0

SGLang

How to use yentinglin/Taiwan-LLaMa-v1.0 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "yentinglin/Taiwan-LLaMa-v1.0" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "yentinglin/Taiwan-LLaMa-v1.0",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "yentinglin/Taiwan-LLaMa-v1.0" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "yentinglin/Taiwan-LLaMa-v1.0",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use yentinglin/Taiwan-LLaMa-v1.0 with Docker Model Runner:
```
docker model run hf.co/yentinglin/Taiwan-LLaMa-v1.0
```

Failure to run inference when running on Gradio

by johannchu - opened Sep 9, 2023

Discussion

johannchu

Sep 9, 2023

Hi all, I had successfully run inference with TaiwanLLaMa by loading the model directly on Jupyter notebook. Nevertheless, when I tried to host the model as an API with Gradio (on NVIDIA A100 GPU * 4), the status wheel would just keep spinning and never provide any output. Since the same code worked with Llama-2-7b-chat-hf, I wonder if there's any architectural change in TaiwanLLaMa that prohibits us from using the same code to get the output. Thanks!

maggie-lee

Sep 13, 2023

I have same question, Thanks!

yentinglin

Owner Sep 28, 2023

No architectural change in TW LLM. Could you provide the gradio scripts?

johannchu

Oct 18, 2023

The script is listed here:
https://github.com/johannchu/taiwan-llama-on-gradio/blob/main/demo.py

To avoid downloading the model every time, we save the model checkpoint and load it locally. For the sake of bandwidth, we did not upload the model to the repo.
Also, to make sure there are no mistake in Gradio app itself, we added a "get_random_asnwer()" function in the script for testing. When we switched to using "get_random_answer()" under "respond()", the web app can display chat content normally. Nevertheless, when we switched to "get_reply_from_llm()" (i.e. the function running LlaMa inference), the web app would just keep spinning and never respond.

What's interesting is that if we use "get_reply_from_llm()" in a script with prompt hardcoded in (instead of hosting the whole thing as an app), the script can function normally, albeit the longer waiting time.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment