Instructions to use Technoculture/Medtulu-2x7b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Technoculture/Medtulu-2x7b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Technoculture/Medtulu-2x7b")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Technoculture/Medtulu-2x7b")
model = AutoModelForCausalLM.from_pretrained("Technoculture/Medtulu-2x7b", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Technoculture/Medtulu-2x7b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Technoculture/Medtulu-2x7b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Technoculture/Medtulu-2x7b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/Technoculture/Medtulu-2x7b

SGLang

How to use Technoculture/Medtulu-2x7b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Technoculture/Medtulu-2x7b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Technoculture/Medtulu-2x7b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Technoculture/Medtulu-2x7b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Technoculture/Medtulu-2x7b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use Technoculture/Medtulu-2x7b with Docker Model Runner:
```
docker model run hf.co/Technoculture/Medtulu-2x7b
```

Medtulu-2x7b / README.md

satyamt

Upload folder using huggingface_hub

4a9d3b8 verified over 2 years ago

preview code

Raw

History Blame Contribute Delete

3.38 kB

metadata

license: apache-2.0
tags:
  - moe
  - merge
  - Technoculture/MT7Bi-dpo
  - allenai/tulu-2-dpo-7b

Medtulu-2x7b

Medtulu-2x7b is a Mixure of Experts (MoE) made with the following models:

🧩 Configuration

base_model: Technoculture/MT7Bi-dpo
tokenizer_source: union
gate_mode: hidden
dtype: bfloat16
experts:
  - source_model: Technoculture/MT7Bi-dpo
    positive_prompts:
      - "Are elevated serum levels of interleukin 21 associated with disease severity in patients with psoriasis?"
      - "Which one of the following does NOT present antigens?"
      - "A 25-year-old male patient presents to your clinic in significant distress. He states he has excruciating, stabbing pain around the left side of his head, and his left eye will not stop tearing. These types of headaches have been occurring for the past week every morning when he awakens and last around 60 minutes. He denies any aura, nausea, or vomiting. He denies any other past medical history. What is this patient's diagnosis?"
      - "When using an inhaler, when should a patient be asked to rinse their mouth?"
      - "What is the embryological origin of the hyoid bone?"
      - "After what period of time does maximal dynamic exercise become predominantly aerobic?"
  - source_model: allenai/tulu-2-dpo-7b
    positive_prompts:
      - "Who composed the tune of 'Twinkle, Twinkle, Little Star'?"
      - "Gem went to get new supplies for her hamster and she found snacks and exercise balls She chose the _ because her hamster was fat."
      - "John orders food for a massive restaurant. He orders 1000 pounds of beef for $8 per pound. He also orders twice that much chicken at $3 per pound. How much did everything cost?"
      - "The gravitational force of the Sun affects the planets in our solar system. Which of these is influenced the most by this force?"
      - "2sin(x) + yz ="
      - "Hobbies and Crafts"

Evaluations

Benchmark	Medtulu-2x7b	Orca-2-7b	llama-2-7b	meditron-7b	meditron-70b
MedMCQA
ClosedPubMedQA
PubMedQA
MedQA
MedQA4
MedicationQA
MMLU Medical
MMLU
TruthfulQA
GSM8K
ARC
HellaSwag
Winogrande

More details on the Open LLM Leaderboard evaluation results can be found here.

💻 Usage

!pip install -qU transformers bitsandbytes accelerate

from transformers import AutoTokenizer
import transformers
import torch

model = "Technoculture/Medtulu-2x7b"

tokenizer = AutoTokenizer.from_pretrained(model)
pipeline = transformers.pipeline(
    "text-generation",
    model=model,
    model_kwargs={"torch_dtype": torch.float16, "load_in_4bit": True},
)

messages = [{"role": "user", "content": "Explain what a Mixture of Experts is in less than 100 words."}]
prompt = pipeline.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
outputs = pipeline(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
print(outputs[0]["generated_text"])