Instructions to use yanolja/KoSOLAR-10.7B-v0.2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use yanolja/KoSOLAR-10.7B-v0.2 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="yanolja/KoSOLAR-10.7B-v0.2")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("yanolja/KoSOLAR-10.7B-v0.2")
model = AutoModelForCausalLM.from_pretrained("yanolja/KoSOLAR-10.7B-v0.2", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use yanolja/KoSOLAR-10.7B-v0.2 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "yanolja/KoSOLAR-10.7B-v0.2"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "yanolja/KoSOLAR-10.7B-v0.2",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/yanolja/KoSOLAR-10.7B-v0.2

SGLang

How to use yanolja/KoSOLAR-10.7B-v0.2 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "yanolja/KoSOLAR-10.7B-v0.2" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "yanolja/KoSOLAR-10.7B-v0.2",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "yanolja/KoSOLAR-10.7B-v0.2" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "yanolja/KoSOLAR-10.7B-v0.2",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use yanolja/KoSOLAR-10.7B-v0.2 with Docker Model Runner:
```
docker model run hf.co/yanolja/KoSOLAR-10.7B-v0.2
```

토크나이저에 이슈가 있었나요?

by StatPan - opened Jan 30, 2024

Discussion

StatPan

Jan 30, 2024

https://huggingface.co/LDCC/LDCC-SOLAR-10.7B

현 1등 모델의 정보를 보던 중에, 이 모델에 토크나이저 이슈가 있다고 언급이 되어 있던데,
간단하게 테스트 해봤을 땐 문제가 없는 것 같아서 문의 남겨봅니다

seungdeok

Jan 30, 2024

The question is if there was a tokenizer issue with the previous version v0.1.
Yes, there was. So we deprecated the previous model.
Tokenizer used in v0.2 has been shown that there is no issue so far. We repeatedly tested this version of the tokenizer and it decoded and encoded tokens well.

StatPan

Jan 30, 2024

•

edited Jan 30, 2024

답장 감사드립니다~!
LDCC 쪽에서 말씀하신 내용은 v0.1에서 확인된 내용이 맞았습니다.
모델 오픈 소스 기여에 감사드리며 모델 잘 써보겠습니다

soonh1618

Feb 7, 2024

tokenizer.model 파일이 v0.2에는 없는데 v0.1에서 다운 받으면 될까요?

seungduk

Yanolja org Feb 7, 2024

tokenizer.model 파일이 v0.2에는 없는데 v0.1에서 다운 받으면 될까요?

아니요, v0.1과는 vocab size부터 다릅니다. tokenizer.model이 없어도 Fast tokenizer를 쓰시면 불러오실 수 있습니다.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment