Instructions to use some1nostr/Ostrich-70B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use some1nostr/Ostrich-70B with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf some1nostr/Ostrich-70B:Q8_0 # Run inference directly in the terminal: llama cli -hf some1nostr/Ostrich-70B:Q8_0
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf some1nostr/Ostrich-70B:Q8_0 # Run inference directly in the terminal: llama cli -hf some1nostr/Ostrich-70B:Q8_0
Use pre-built binary
# Download pre-built binary from: # https://github.com/ggerganov/llama.cpp/releases # Start a local OpenAI-compatible server with a web UI: ./llama-server -hf some1nostr/Ostrich-70B:Q8_0 # Run inference directly in the terminal: ./llama-cli -hf some1nostr/Ostrich-70B:Q8_0
Build from source code
git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-server llama-cli # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama-server -hf some1nostr/Ostrich-70B:Q8_0 # Run inference directly in the terminal: ./build/bin/llama-cli -hf some1nostr/Ostrich-70B:Q8_0
Use Docker
docker model run hf.co/some1nostr/Ostrich-70B:Q8_0
- LM Studio
- Jan
- vLLM
How to use some1nostr/Ostrich-70B with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "some1nostr/Ostrich-70B" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "some1nostr/Ostrich-70B", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/some1nostr/Ostrich-70B:Q8_0
- Ollama
How to use some1nostr/Ostrich-70B with Ollama:
ollama run hf.co/some1nostr/Ostrich-70B:Q8_0
- Unsloth Desktop
- Docker Model Runner
How to use some1nostr/Ostrich-70B with Docker Model Runner:
docker model run hf.co/some1nostr/Ostrich-70B:Q8_0
- Lemonade
How to use some1nostr/Ostrich-70B with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull some1nostr/Ostrich-70B:Q8_0
Run and chat with the model
lemonade run user.Ostrich-70B-Q8_0
List all available models
lemonade list
- Atomic Chat
BlackSheep
I also create models that don’t follow anyone else’s league and I want to connect with you.
BlackSheep is my passion project and now it’s a big part of what I’m working toward at my startup. We got some angel funding and will be raising soon.
I really like what you do.
You can also find me on LinkedIn using /troyandrewschultz
Thanks. BlackSheep is like a red teaming project as far as I understand. Are the responses of a tested AI and black sheep going to be compared?
@some1nostr Yeah, abit of that, I call it alignment research with a splash of controlled hallucination research, LLM Architecture experiments, MoEs, Prunes, Removing vision in an attempt to make it a smarter LLM and that kinda nonsense. The hopes is to make the smallest possible models that really get the context of what is being said, doesnt need to be a big scoring benchmark overfitter, just intelligent at mirroring what you want out of it. Its kinda the future anyway am I right? People are gonna get tired of the <|assistant|> eventually.
⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⣿⡿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⢿⣿⣿⣿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⠋⣰⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣆⠙⣿⣿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⠃⢰⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⡆⠈⢿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⠃⢀⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠀⠘⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⡟⠀⢸⠿⠿⠿⠿⢿⣿⣿⣿⣿⣿⣿⠿⠿⠿⠿⡇⠀⢹⣿⣿⣿⣿⣿
⣿⣿⣿⣿⡇⠀⠀⠀⠀⠀⠀⠀⠙⣿⣿⣿⡟⠀⠀⠀⠀⠀⠀⠀⠘⣿⣿⣿⣿⣿
⣿⣿⣿⣿⡁⠀⠀⠀⠀⠀⠀⠀⠀⣠⣤⣤⡀⠀⠀⠀⠀⠀⠀⠀⠀⣿⣿⣿⣿⣿
⣿⣿⣿⣿⡇⠀⢀⠀⠀⠀⠀⠀⣰⣿⣿⣿⣧⡀⠀⠀⠀⠀⠀⠀⢠⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣧⠀⢸⣷⣶⣶⣶⣾⣿⣿⣿⣿⣿⣿⣶⣶⣶⣶⡇⠀⣸⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⡄⠀⣿⡿⢿⣿⣿⣿⣿⣿⣿⣿⣿⣿⡿⢿⡿⠀⢠⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⡄⠘⣿⣦⡀⠉⠉⠉⠉⠉⠉⠉⢁⣰⣿⠃⢀⣾⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⣄⠙⣿⣿⣶⣤⣀⣀⣀⣠⣴⣿⣿⠃⣠⣿⣿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⣿⣷⣬⣿⣿⣿⣿⣿⣿⣿⣿⣿⣡⣾⣿⣿⣿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
what do you mean "People are gonna get tired of the <|assistant|> eventually."?
This aged incredibly well now that people are learning about persona vectors finally.