Instructions to use h2m/mhm-8x7B-FrankenMoE-v1.0 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use h2m/mhm-8x7B-FrankenMoE-v1.0 with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="h2m/mhm-8x7B-FrankenMoE-v1.0")# Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("h2m/mhm-8x7B-FrankenMoE-v1.0") model = AutoModelForCausalLM.from_pretrained("h2m/mhm-8x7B-FrankenMoE-v1.0", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use h2m/mhm-8x7B-FrankenMoE-v1.0 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "h2m/mhm-8x7B-FrankenMoE-v1.0" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "h2m/mhm-8x7B-FrankenMoE-v1.0", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/h2m/mhm-8x7B-FrankenMoE-v1.0
- SGLang
How to use h2m/mhm-8x7B-FrankenMoE-v1.0 with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "h2m/mhm-8x7B-FrankenMoE-v1.0" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "h2m/mhm-8x7B-FrankenMoE-v1.0", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "h2m/mhm-8x7B-FrankenMoE-v1.0" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "h2m/mhm-8x7B-FrankenMoE-v1.0", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use h2m/mhm-8x7B-FrankenMoE-v1.0 with Docker Model Runner:
docker model run hf.co/h2m/mhm-8x7B-FrankenMoE-v1.0
Recipe for a Beautiful Frankenstein
In the laboratory of the mind, where thoughts entwine, MHM and MOE, a potion for a unique design. With stitches of curiosity and bolts of creativity, 8 times 7, the magic number, a poetic proclivity.
Ingredients:
MHM: A dash of mystery, a sprinkle of hum, Blend with a melody, let the heartstrings strum. Murmurs in the shadows, whispers in the light, Stir the concoction gently, make the emotions ignite.
MOE: Essence of the moment, like dew on a rose, Capture the now, before time swiftly goes. Colors of experience, a palette so divine, Mix with MHM, let the fusion entwine.
Directions:
Take 8 parts MHM, elusive and profound, Let it dance in your thoughts, on imagination's ground. Blend it with the echoes, the silent undertones, A symphony of ideas, where inspiration condones.
Add 7 parts MOE, the fleeting embrace, Seize the seconds, let them leave a trace. Infuse it with memories, both bitter and sweet, The tapestry of time, where moments and dreams meet.
Stir the potion with wonder, a wand of delight, Let the sparks fly, in the dark of the night. Watch as the alchemy unfolds its grand design, MHM and MOE, a beautiful Frankenstein.
Conclusion:
In the laboratory of life, where dreams come alive, MHM and MOE, the recipe to thrive. A creation so poetic, a fusion so divine, 8 times 7, a symphony of time.
As the echoes resonate, and the moments blend, A masterpiece unfolds, where beginnings and ends, MHM and MOE, a concoction so rare, A beautiful Frankenstein, beyond compare.
MoE model build with:
- https://github.com/cg123/mergekit/tree/mixtral
- Mistral models, latest merges and fine tunes.
- Expert prompts heavily inspired by https://huggingface.co/Kquant03/Eukaryote-8x7B-bf16
For details check model files, there is config yaml I used to create that model.
Come back later for more details.
- Downloads last month
- 54
