Instructions to use brucethemoose/CapyTessBorosYi-34B-200K-DARE-Ties with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use brucethemoose/CapyTessBorosYi-34B-200K-DARE-Ties with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="brucethemoose/CapyTessBorosYi-34B-200K-DARE-Ties")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("brucethemoose/CapyTessBorosYi-34B-200K-DARE-Ties")
model = AutoModelForCausalLM.from_pretrained("brucethemoose/CapyTessBorosYi-34B-200K-DARE-Ties", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use brucethemoose/CapyTessBorosYi-34B-200K-DARE-Ties with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "brucethemoose/CapyTessBorosYi-34B-200K-DARE-Ties"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "brucethemoose/CapyTessBorosYi-34B-200K-DARE-Ties",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/brucethemoose/CapyTessBorosYi-34B-200K-DARE-Ties

SGLang

How to use brucethemoose/CapyTessBorosYi-34B-200K-DARE-Ties with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "brucethemoose/CapyTessBorosYi-34B-200K-DARE-Ties" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "brucethemoose/CapyTessBorosYi-34B-200K-DARE-Ties",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "brucethemoose/CapyTessBorosYi-34B-200K-DARE-Ties" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "brucethemoose/CapyTessBorosYi-34B-200K-DARE-Ties",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use brucethemoose/CapyTessBorosYi-34B-200K-DARE-Ties with Docker Model Runner:
```
docker model run hf.co/brucethemoose/CapyTessBorosYi-34B-200K-DARE-Ties
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

CaPlatTessDolXaBoros-Yi-34B-200K-DARE-Ties-HighDensity

NousResearch/Nous-Capybara-34B, migtissera/Tess-M-v1.3 and bhenrym14/airoboros-3_1-yi-34b-200k merged with a new, experimental implementation of "dare ties" via mergekit. See:

Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch

https://github.com/yule-BUAA/MergeLM

https://github.com/cg123/mergekit/tree/dare'

Merged with the following config, and the tokenizer from chargoddard's Yi-Llama:

models:
  - model: /home/alpha/Storage/Models/Raw/chargoddard_Yi-34B-200K-Llama
    # no parameters necessary for base model
  - model: /home/alpha/Storage/Models/Raw/migtissera_Tess-M-v1.3
    parameters:
      weight: 0.41
      density: 0.50
  - model: /home/alpha//Storage/Models/Raw/bhenrym14_airoboros-3_1-yi-34b-200k
    parameters:
      weight: 0.18
      density: 0.46
  - model: /home/alpha/Storage/Models/Raw/Nous-Capybara-34B
    parameters:
      weight: 0.41
      density: 0.50
merge_method: dare_ties
base_model: /home/alpha/Storage/Models/Raw/chargoddard_Yi-34B-200K-Llama
parameters:
int8_mask: true
dtype: bfloat16

dare_ties is testing with better perplexity than a regular ties merge with the same merge configuration. Model weights that add up to one also seem optimal from testing. And high context results seem... better than the previous dare merge with Tess 1.2.

I chose not to include other finetunes, such as Dolphin, because they aren't trained on the 200K base. If any other 200K finetunes pop up, let me know.

Prompt template: Orca-Vicuna

SYSTEM: {system_message}
USER: {prompt}
ASSISTANT:

Being a Yi model, try disabling the BOS token and/or running a lower temperature with MinP (and no other samplers) if output doesn't seem right. Yi tends to run "hot" by default.

Sometimes the model "spells out" the stop token as </s> like Capybara, so you may need to add </s> as an additional stopping condition. It also might respond to the llama-2 chat format.

24GB GPUs can run Yi-34B-200K models at 45K-75K context with exllamav2. I go into more detail in this post, and recommend exl2 quantizations on data similar to the desired task, such as these targeted at story writing: 4.0bpw / 3.1bpw

Credits:

https://github.com/cg123/mergekit/tree/dare

https://huggingface.co/NousResearch/Nous-Capybara-34B/

https://huggingface.co/bhenrym14/airoboros-3_1-yi-34b-200k

https://huggingface.co/migtissera/Tess-M-v1.3

https://huggingface.co/chargoddard/Yi-34B-200K-Llama

https://huggingface.co/01-ai/Yi-34B-200K

Downloads last month: 67

Safetensors

Model size

34B params

Tensor type

BF16

Model tree for brucethemoose/CapyTessBorosYi-34B-200K-DARE-Ties

Quantizations

4 models