Instructions to use shisa-ai/shisa-v1-llama3-70b.2e5 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use shisa-ai/shisa-v1-llama3-70b.2e5 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="shisa-ai/shisa-v1-llama3-70b.2e5")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("shisa-ai/shisa-v1-llama3-70b.2e5")
model = AutoModelForCausalLM.from_pretrained("shisa-ai/shisa-v1-llama3-70b.2e5", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use shisa-ai/shisa-v1-llama3-70b.2e5 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "shisa-ai/shisa-v1-llama3-70b.2e5"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "shisa-ai/shisa-v1-llama3-70b.2e5",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/shisa-ai/shisa-v1-llama3-70b.2e5

SGLang

How to use shisa-ai/shisa-v1-llama3-70b.2e5 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "shisa-ai/shisa-v1-llama3-70b.2e5" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "shisa-ai/shisa-v1-llama3-70b.2e5",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "shisa-ai/shisa-v1-llama3-70b.2e5" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "shisa-ai/shisa-v1-llama3-70b.2e5",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use shisa-ai/shisa-v1-llama3-70b.2e5 with Docker Model Runner:
```
docker model run hf.co/shisa-ai/shisa-v1-llama3-70b.2e5
```

shisa-v1-llama3-70b.2e5 / README.md

leonardlin

Update README.md

3cc257f verified over 2 years ago

preview code

Raw

History Blame

5.51 kB

	---
	license: llama3
	datasets:
	- augmxnt/ultra-orca-boros-en-ja-v1
	language:
	- ja
	- en
	base_model: meta-llama/Meta-Llama-3-70B-Instruct
	tags:
	- axolotl
	- generated_from_trainer
	model-index:
	- name: shisa-llama3-70b-v1
	results: []
	---

	shisa-v2 Base Model ablation

	The 8e-6 version is better and you should probably use that one.

	Using a [fork](https://github.com/shisa-ai/shaberi) of [Lightblue's Shaberi benchmark framework](https://github.com/lightblue-tech/japanese_llm_eval):

	\| Model \| Average \| ELYZA-tasks-100 \| MT-Bench \| Rakuda \| Tengu-Bench \|
	\|----------------------------------------\|---------\|-----------------\|----------\|--------\|-------------\|
	\| gpt-4-turbo-2024-04-09 \| 8.75 \| 8.78 \| 8.74 \| 9.18 \| 8.31 \|
	\| CohereForAI/c4ai-command-r-plus \| 7.69 \| 7.50 \| 7.43 \| 9.05 \| 6.79 \|
	\| gpt-3.5-turbo-0125 \| 7.17 \| 7.24 \| 6.98 \| 7.64 \| 6.82 \|
	\| shisa-ai/shisa-v1-llama3-70b \| 7.17\| 7.16 \| 7.45 \| 7.98 \| 6.09 \|
	\| karakuri-ai/karakuri-lm-70b-chat-v0.1 \| 6.84 \| 6.86 \| 6.43 \| 7.85 \| 6.23 \|
	\| lightblue/ao-karasu-72B \| 6.81 \| 7.19 \| 6.54 \| 7.25 \| 6.27 \|
	\| shisa-ai/shisa-v1-llama3-8b^ \| 6.29\| 6.62 \| 6.41 \| 7.05\|5.07 \|
	\| shisa-ai/shisa-swallowmx-13a47b-v1 \| 6.17 \| 6.48 \| 6.07 \| 7.11 \| 5.03 \|
	\| shisa-ai/shisa-v1-llama3-8b \| 6.10\| 6.52 \| 6.20 \| 6.37\|5.33 \|
	\| Rakuten/RakutenAI-7B-chat \| 5.58 \| 5.92 \| 4.60 \| 6.58 \| 5.24 \|
	\| shisa-ai/shisa-v1-gemma-8b \| 5.64 \| 6.50 \| 5.42 \| 5.10 \| 5.55 \|
	\| augmxnt/shisa-gamma-7b-v1 \| 5.56 \| 5.84 \| 4.00 \| 6.73 \| 5.68 \|
	\| lightblue/qarasu-14B-chat-plus-unleashed \| 5.20 \| 5.58 \| 4.74 \| 5.46 \| 5.01 \|
	\| cyberagent/calm2-7b-chat \| 4.76 \| 4.90 \| 3.58 \| 5.75 \| 4.81 \|
	\| mistralai/Mistral-7B-Instruct-v0.2 \| 4.69 \| 5.78 \| 4.65 \| 3.80 \| 4.53 \|
	\| shisa-ai/shisa-v1-yi1.5-9b \| 4.63\| 5.98 \| 4.28 \| 3.26\|5.00 \|


	<!-- This model card has been generated automatically according to the information the Trainer had access to. You
	should probably proofread and complete it, then remove this comment. -->

	[<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/OpenAccess-AI-Collective/axolotl)
	<details><summary>See axolotl config</summary>

	axolotl version: `0.4.0`
	```yaml
	base_model: meta-llama/Meta-Llama-3-70B-Instruct
	model_type: LlamaForCausalLM
	tokenizer_type: AutoTokenizer

	load_in_8bit: false
	load_in_4bit: false
	strict: false

	hub_model_id: shisa-ai/shisa-llama3-70b-v1
	hub_strategy: end

	use_wandb: true
	wandb_project: shisa-v2
	wandb_entity: augmxnt
	wandb_name: shisa-llama3-70b-v1

	chat_template: llama3
	datasets:
	- path: augmxnt/ultra-orca-boros-en-ja-v1
	type: sharegpt
	dataset_prepared_path: last_run_prepared
	val_set_size: 0.05
	output_dir: ./outputs/basemodel-llama3-70b

	sequence_len: 4096
	sample_packing: true
	pad_to_sequence_len: true

	gradient_accumulation_steps: 2
	micro_batch_size: 2
	num_epochs: 3
	optimizer: paged_adamw_8bit
	lr_scheduler: linear
	learning_rate: 2e-5

	train_on_inputs: false
	group_by_length: false
	bf16: auto
	fp16:
	tf32: true

	gradient_checkpointing: true
	gradient_checkpointing_kwargs:
	use_reentrant: false
	early_stopping_patience:
	resume_from_checkpoint:
	logging_steps: 1
	xformers_attention:
	flash_attention: true

	warmup_ratio: 0.1
	evals_per_epoch: 2
	eval_table_size:
	saves_per_epoch: 0
	debug:
	deepspeed: axolotl/deepspeed_configs/zero3_bf16.json
	weight_decay: 0.05
	fsdp:
	fsdp_config:
	special_tokens:
	pad_token: <\|end_of_text\|>

	```

	</details><br>

	# shisa-llama3-70b-v1

	This model is a fine-tuned version of [meta-llama/Meta-Llama-3-70B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct) on the None dataset.
	It achieves the following results on the evaluation set:
	- Loss: 0.4425

	## Model description

	More information needed

	## Intended uses & limitations

	More information needed

	## Training and evaluation data

	More information needed

	## Training procedure

	### Training hyperparameters

	The following hyperparameters were used during training:
	- learning_rate: 2e-05
	- train_batch_size: 2
	- eval_batch_size: 2
	- seed: 42
	- distributed_type: multi-GPU
	- num_devices: 16
	- gradient_accumulation_steps: 2
	- total_train_batch_size: 64
	- total_eval_batch_size: 32
	- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
	- lr_scheduler_type: linear
	- lr_scheduler_warmup_steps: 87
	- num_epochs: 3

	### Training results


	\| Training Loss \| Epoch \| Step \| Validation Loss \|
	\|:-------------:\|:------:\|:----:\|:---------------:\|
	\| 1.2478 \| 0.0033 \| 1 \| 0.7102 \|
	\| 0.7516 \| 0.5008 \| 154 \| 0.4325 \|
	\| 0.7185 \| 1.0016 \| 308 \| 0.3966 \|
	\| 0.3708 \| 1.4862 \| 462 \| 0.3976 \|
	\| 0.3758 \| 1.9870 \| 616 \| 0.3840 \|
	\| 0.0928 \| 2.4699 \| 770 \| 0.4425 \|


	### Framework versions

	- Transformers 4.40.2
	- Pytorch 2.3.0+cu121
	- Datasets 2.19.1
	- Tokenizers 0.19.1