Instructions to use Weyaxi/Einstein-v6-7B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Weyaxi/Einstein-v6-7B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Weyaxi/Einstein-v6-7B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Weyaxi/Einstein-v6-7B")
model = AutoModelForCausalLM.from_pretrained("Weyaxi/Einstein-v6-7B", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Weyaxi/Einstein-v6-7B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Weyaxi/Einstein-v6-7B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Weyaxi/Einstein-v6-7B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Weyaxi/Einstein-v6-7B

SGLang

How to use Weyaxi/Einstein-v6-7B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Weyaxi/Einstein-v6-7B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Weyaxi/Einstein-v6-7B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Weyaxi/Einstein-v6-7B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Weyaxi/Einstein-v6-7B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Weyaxi/Einstein-v6-7B with Docker Model Runner:
```
docker model run hf.co/Weyaxi/Einstein-v6-7B
```

Einstein-v6-7B / README.md

Weyaxi

Adding AWQ quants (#5)

71fbffe verified over 2 years ago

preview code

Raw

History Blame Contribute Delete

10.1 kB

	---
	language:
	- en
	license: other
	tags:
	- axolotl
	- generated_from_trainer
	- Mistral
	- instruct
	- finetune
	- chatml
	- gpt4
	- synthetic data
	- science
	- physics
	- chemistry
	- biology
	- math
	base_model: alpindale/Mistral-7B-v0.2-hf
	datasets:
	- allenai/ai2_arc
	- camel-ai/physics
	- camel-ai/chemistry
	- camel-ai/biology
	- camel-ai/math
	- metaeval/reclor
	- openbookqa
	- mandyyyyii/scibench
	- derek-thomas/ScienceQA
	- TIGER-Lab/ScienceEval
	- jondurbin/airoboros-3.2
	- LDJnr/Capybara
	- Cot-Alpaca-GPT4-From-OpenHermes-2.5
	- STEM-AI-mtl/Electrical-engineering
	- knowrohit07/saraswati-stem
	- sablo/oasst2_curated
	- lmsys/lmsys-chat-1m
	- TIGER-Lab/MathInstruct
	- bigbio/med_qa
	- meta-math/MetaMathQA-40K
	- openbookqa
	- piqa
	- metaeval/reclor
	- derek-thomas/ScienceQA
	- scibench
	- sciq
	- Open-Orca/SlimOrca
	- migtissera/Synthia-v1.3
	- TIGER-Lab/ScienceEval
	- allenai/WildChat
	- microsoft/orca-math-word-problems-200k
	- openchat/openchat_sharegpt4_dataset
	- teknium/GPTeacher-General-Instruct
	- m-a-p/CodeFeedback-Filtered-Instruction
	- totally-not-an-llm/EverythingLM-data-V3
	- HuggingFaceH4/no_robots
	- OpenAssistant/oasst_top1_2023-08-25
	- WizardLM/WizardLM_evol_instruct_70k
	model-index:
	- name: Einstein-v6-7B
	results:
	- task:
	type: text-generation
	name: Text Generation
	dataset:
	name: AI2 Reasoning Challenge (25-Shot)
	type: ai2_arc
	config: ARC-Challenge
	split: test
	args:
	num_few_shot: 25
	metrics:
	- type: acc_norm
	value: 63.57
	name: normalized accuracy
	source:
	url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Weyaxi/Einstein-v6-7B
	name: Open LLM Leaderboard
	- task:
	type: text-generation
	name: Text Generation
	dataset:
	name: HellaSwag (10-Shot)
	type: hellaswag
	split: validation
	args:
	num_few_shot: 10
	metrics:
	- type: acc_norm
	value: 82.76
	name: normalized accuracy
	source:
	url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Weyaxi/Einstein-v6-7B
	name: Open LLM Leaderboard
	- task:
	type: text-generation
	name: Text Generation
	dataset:
	name: MMLU (5-Shot)
	type: cais/mmlu
	config: all
	split: test
	args:
	num_few_shot: 5
	metrics:
	- type: acc
	value: 62.23
	name: accuracy
	source:
	url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Weyaxi/Einstein-v6-7B
	name: Open LLM Leaderboard
	- task:
	type: text-generation
	name: Text Generation
	dataset:
	name: TruthfulQA (0-shot)
	type: truthful_qa
	config: multiple_choice
	split: validation
	args:
	num_few_shot: 0
	metrics:
	- type: mc2
	value: 52.02
	source:
	url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Weyaxi/Einstein-v6-7B
	name: Open LLM Leaderboard
	- task:
	type: text-generation
	name: Text Generation
	dataset:
	name: Winogrande (5-shot)
	type: winogrande
	config: winogrande_xl
	split: validation
	args:
	num_few_shot: 5
	metrics:
	- type: acc
	value: 78.61
	name: accuracy
	source:
	url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Weyaxi/Einstein-v6-7B
	name: Open LLM Leaderboard
	- task:
	type: text-generation
	name: Text Generation
	dataset:
	name: GSM8k (5-shot)
	type: gsm8k
	config: main
	split: test
	args:
	num_few_shot: 5
	metrics:
	- type: acc
	value: 63.53
	name: accuracy
	source:
	url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Weyaxi/Einstein-v6-7B
	name: Open LLM Leaderboard
	---
	![image/png](https://cdn-uploads.huggingface.co/production/uploads/6468ce47e134d050a58aa89c/CxDk4KKhQqL-Pg0AMn1gb.png)

	# 🔬 Einstein-v6-7B

	This model is a full fine-tuned version of [alpindale/Mistral-7B-v0.2-hf](https://huggingface.co/alpindale/Mistral-7B-v0.2-hf) on diverse datasets.

	This model is finetuned using `8xRTX3090` + `1xRTXA6000` using [axolotl](https://github.com/OpenAccess-AI-Collective/axolotl).

	This model's training was sponsored by [sablo.ai](https://sablo.ai).

	<details><summary>See axolotl config</summary>

	axolotl version: `0.4.0`
	```yaml
	base_model: alpindale/Mistral-7B-v0.2-hf
	model_type: MistralForCausalLM
	tokenizer_type: LlamaTokenizer
	is_mistral_derived_model: true

	load_in_8bit: false
	load_in_4bit: false
	strict: false

	chat_template: chatml
	datasets:
	- path: data/merged_all.json
	ds_type: json
	type: alpaca
	conversation: chatml

	- path: data/gpteacher-instruct-special-alpaca.json
	ds_type: json
	type: gpteacher
	conversation: chatml

	- path: data/wizardlm_evol_instruct_70k_random_half.json
	ds_type: json
	type: alpaca
	conversation: chatml

	- path: data/capybara_sharegpt.json
	ds_type: json
	type: sharegpt
	conversation: chatml

	- path: data/synthia-v1.3_sharegpt_12500.json
	ds_type: json
	type: sharegpt
	conversation: chatml

	- path: data/cot_alpaca_gpt4_extracted_openhermes_2.5_sharegpt.json
	ds_type: json
	type: sharegpt
	conversation: chatml

	- path: data/slimorca_dedup_filtered_95k_sharegpt.json
	ds_type: json
	type: sharegpt
	conversation: chatml

	- path: data/airoboros_3.2_without_contextual_slimorca_orca_sharegpt.json
	ds_type: json
	type: sharegpt
	conversation: chatml

	- path: data/allenai_wild_chat_gpt4_english_toxic_random_half_4k_sharegpt.json
	ds_type: json
	type: sharegpt
	strict: false
	conversation: chatml

	- path: data/pippa_bagel_repo_3k_sharegpt.json
	ds_type: json
	type: sharegpt
	conversation: chatml

	- path: data/gpt4_data_lmys_1m_sharegpt.json
	ds_type: json
	type: sharegpt
	conversation: chatml

	- path: data/sharegpt_gpt4_english.json
	ds_type: json
	type: sharegpt
	conversation: chatml

	- path: data/no_robots_sharegpt.json
	ds_type: json
	type: sharegpt
	strict: false
	conversation: chatml

	- path: data/oasst_top1_from_fusechatmixture_sharegpt.json
	ds_type: json
	type: sharegpt
	strict: false
	conversation: chatml

	- path: data/everythinglm-data-v3_sharegpt.json
	ds_type: json
	type: sharegpt
	strict: false
	conversation: chatml

	dataset_prepared_path: last_run_prepared
	# val_set_size: 0.005
	val_set_size: 0.0

	do_bench_eval: true

	output_dir: ./Einstein-v6-7B-model

	sequence_len: 8192
	sample_packing: true
	pad_to_sequence_len: true
	eval_sample_packing: false

	wandb_project: Einstein
	wandb_entity:
	wandb_watch:
	wandb_name:
	wandb_log_model:
	hub_model_id: Weyaxi/Einstein-v6-7B

	save_safetensors: true

	gradient_accumulation_steps: 4
	micro_batch_size: 1
	num_epochs: 2
	optimizer: adamw_bnb_8bit
	lr_scheduler: cosine
	learning_rate: 0.000005

	train_on_inputs: false
	group_by_length: false
	bf16: true
	fp16: false
	tf32: false

	gradient_checkpointing: true
	early_stopping_patience:
	resume_from_checkpoint:
	local_rank:
	logging_steps: 1
	xformers_attention:
	flash_attention: true

	warmup_steps: 10
	evals_per_epoch: 3 # changed
	eval_table_size:
	eval_table_max_new_tokens: 128
	saves_per_epoch: 2 # changed
	debug:

	deepspeed: zero3_bf16.json
	weight_decay: 0.0
	fsdp:
	fsdp_config:
	special_tokens:
	bos_token: "<s>"
	eos_token: "<\|im_end\|>"
	unk_token: "<unk>"
	tokens:
	- "<\|im_start\|>"
	```

	</details><br>

	# 💬 Prompt Template

	You can use this prompt template while using the model:

	### ChatML

	```
	<\|im_start\|>system
	{system}<\|im_end\|>
	<\|im_start\|>user
	{user}<\|im_end\|>
	<\|im_start\|>assistant
	{asistant}<\|im_end\|>
	```

	This prompt template is available as a [chat template](https://huggingface.co/docs/transformers/main/chat_templating), which means you can format messages using the
	`tokenizer.apply_chat_template()` method:

	```python
	messages = [
	{"role": "system", "content": "You are helpful AI asistant."},
	{"role": "user", "content": "Hello!"}
	]
	gen_input = tokenizer.apply_chat_template(message, return_tensors="pt")
	model.generate(**gen_input)
	```

	# 🔄 Quantizationed versions

	## GGUF [@bartowski](https://huggingface.co/bartowski)

	- https://huggingface.co/bartowski/Einstein-v6-7B-GGUF

	## ExLlamaV2 [@bartowski](https://huggingface.co/bartowski)

	- https://huggingface.co/bartowski/Einstein-v6-7B-exl2

	## AWQ [@solidrust](https://huggingface.co/solidrust)

	- https://huggingface.co/solidrust/Einstein-v6-7B-AWQ

	# 🎯 [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
	Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_Weyaxi__Einstein-v6-7B)

	\| Metric \|Value\|
	\|---------------------------------\|----:\|
	\|Avg. \|67.12\|
	\|AI2 Reasoning Challenge (25-Shot)\|63.57\|
	\|HellaSwag (10-Shot) \|82.76\|
	\|MMLU (5-Shot) \|62.23\|
	\|TruthfulQA (0-shot) \|52.02\|
	\|Winogrande (5-shot) \|78.61\|
	\|GSM8k (5-shot) \|63.53\|

	# 🤖 Additional information about training

	This model is full fine-tuned for 2 epoch.

	Total number of steps was 2412.

	<details><summary>Loss graph</summary>

	![image/png](https://cdn-uploads.huggingface.co/production/uploads/6468ce47e134d050a58aa89c/WsZYN8JHUy5iqPUa-xSdJ.png)

	</details><br>

	# 🤝 Acknowledgments

	Thanks to [sablo.ai](https://sablo.ai) for sponsoring this model.

	Thanks to all the dataset authors mentioned in the datasets section.

	Thanks to [axolotl](https://github.com/OpenAccess-AI-Collective/axolotl) for making the repository I used to make this model.

	Thanks to all open source AI community.

	[<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/OpenAccess-AI-Collective/axolotl)

	If you would like to support me:

	[☕ Buy Me a Coffee](https://www.buymeacoffee.com/weyaxi)