Text Generation
Transformers
Safetensors
MLX
English
nemotron-nas
nvidia
unsloth - llama-3 - pytorch
mlx-my-repo
conversational
custom_code
4-bit precision
Instructions to use Wwayu/Llama-3_3-Nemotron-Super-49B-v1_5-mlx-4Bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Wwayu/Llama-3_3-Nemotron-Super-49B-v1_5-mlx-4Bit with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="Wwayu/Llama-3_3-Nemotron-Super-49B-v1_5-mlx-4Bit", trust_remote_code=True) messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# Load model directly from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("Wwayu/Llama-3_3-Nemotron-Super-49B-v1_5-mlx-4Bit", trust_remote_code=True, device_map="auto") - MLX
How to use Wwayu/Llama-3_3-Nemotron-Super-49B-v1_5-mlx-4Bit with MLX:
# Make sure mlx-lm is installed # pip install --upgrade mlx-lm # Generate text with mlx-lm from mlx_lm import load, generate model, tokenizer = load("Wwayu/Llama-3_3-Nemotron-Super-49B-v1_5-mlx-4Bit") prompt = "Write a story about Einstein" messages = [{"role": "user", "content": prompt}] prompt = tokenizer.apply_chat_template( messages, add_generation_prompt=True ) text = generate(model, tokenizer, prompt=prompt, verbose=True) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- vLLM
How to use Wwayu/Llama-3_3-Nemotron-Super-49B-v1_5-mlx-4Bit with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "Wwayu/Llama-3_3-Nemotron-Super-49B-v1_5-mlx-4Bit" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Wwayu/Llama-3_3-Nemotron-Super-49B-v1_5-mlx-4Bit", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/Wwayu/Llama-3_3-Nemotron-Super-49B-v1_5-mlx-4Bit
- SGLang
How to use Wwayu/Llama-3_3-Nemotron-Super-49B-v1_5-mlx-4Bit with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "Wwayu/Llama-3_3-Nemotron-Super-49B-v1_5-mlx-4Bit" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Wwayu/Llama-3_3-Nemotron-Super-49B-v1_5-mlx-4Bit", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "Wwayu/Llama-3_3-Nemotron-Super-49B-v1_5-mlx-4Bit" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Wwayu/Llama-3_3-Nemotron-Super-49B-v1_5-mlx-4Bit", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Pi
How to use Wwayu/Llama-3_3-Nemotron-Super-49B-v1_5-mlx-4Bit with Pi:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "Wwayu/Llama-3_3-Nemotron-Super-49B-v1_5-mlx-4Bit"
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "mlx-lm": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "Wwayu/Llama-3_3-Nemotron-Super-49B-v1_5-mlx-4Bit" } ] } } }Run Pi
# Start Pi in your project directory: pi
- MLX LM
How to use Wwayu/Llama-3_3-Nemotron-Super-49B-v1_5-mlx-4Bit with MLX LM:
Generate or start a chat session
# Install MLX LM uv tool install mlx-lm # Interactive chat REPL mlx_lm.chat --model "Wwayu/Llama-3_3-Nemotron-Super-49B-v1_5-mlx-4Bit"
Run an OpenAI-compatible server
# Install MLX LM uv tool install mlx-lm # Start the server mlx_lm.server --model "Wwayu/Llama-3_3-Nemotron-Super-49B-v1_5-mlx-4Bit" # Calling the OpenAI-compatible server with curl curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Wwayu/Llama-3_3-Nemotron-Super-49B-v1_5-mlx-4Bit", "messages": [ {"role": "user", "content": "Hello"} ] }' - Docker Model Runner
How to use Wwayu/Llama-3_3-Nemotron-Super-49B-v1_5-mlx-4Bit with Docker Model Runner:
docker model run hf.co/Wwayu/Llama-3_3-Nemotron-Super-49B-v1_5-mlx-4Bit
- Hermes Agent
How to use Wwayu/Llama-3_3-Nemotron-Super-49B-v1_5-mlx-4Bit with Hermes Agent:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "Wwayu/Llama-3_3-Nemotron-Super-49B-v1_5-mlx-4Bit"
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default Wwayu/Llama-3_3-Nemotron-Super-49B-v1_5-mlx-4Bit
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use Wwayu/Llama-3_3-Nemotron-Super-49B-v1_5-mlx-4Bit with OpenClaw:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "Wwayu/Llama-3_3-Nemotron-Super-49B-v1_5-mlx-4Bit"
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id "Wwayu/Llama-3_3-Nemotron-Super-49B-v1_5-mlx-4Bit" \ --custom-provider-id mlx-lm \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Upload chat_template.jinja with huggingface_hub
Browse files- chat_template.jinja +20 -0
chat_template.jinja
ADDED
|
@@ -0,0 +1,20 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{% set bos = "<|begin_of_text|>" %}{%- set enable_thinking = true -%}{% set system_start_header = "<|start_header_id|>" %}{% set system_end_header = "<|end_header_id|>
|
| 2 |
+
|
| 3 |
+
" %}{% set start_header = "<|start_header_id|>" %}{% set end_header = "<|end_header_id|>
|
| 4 |
+
|
| 5 |
+
" %}{% set eot = "<|eot_id|>" %}{% set system_token = "system" %}{% set user_token = "user" %}{% set assistant_token = "assistant" %}{% set tool_token = "tool" %}{{- bos ~ system_start_header ~ system_token ~ system_end_header -}}{%- if messages[0].role == 'system' and messages[0].content != '' -%}{%- set system_content = messages[0].content -%}{%- if '/no_think' in system_content -%}{%- set system_content = system_content.replace('/no_think', '')|trim -%}{%- set enable_thinking = false -%}{%- elif '/think' in system_content -%}{%- set system_content = system_content.replace('/think', '')|trim -%}{%- set enable_thinking = true -%}{%- endif -%}{{- system_content + '
|
| 6 |
+
|
| 7 |
+
' -}}{%- endif -%}{%- if tools -%}{{- 'You can use the following tools to assist the user if required:
|
| 8 |
+
<AVAILABLE_TOOLS>[' -}}{%- for tool in tools -%}{{- (tool.function if tool.function is defined else tool) | tojson -}}{{- ', ' if not loop.last else '' -}}{%- endfor -%}{{- ']</AVAILABLE_TOOLS>
|
| 9 |
+
|
| 10 |
+
If you decide to call any tool(s), use the following format:
|
| 11 |
+
<TOOLCALL>[{{"name": "tool_name1", "arguments": "tool_args1"}}, {{"name": "tool_name2", "arguments": "tool_args2"}}]</TOOLCALL>
|
| 12 |
+
|
| 13 |
+
Response from tool(s) will be returned in this format:
|
| 14 |
+
<TOOL_RESPONSE>[{{"response": "tool_response1"}}, {{"response": "tool_response2"}}]</TOOL_RESPONSE>
|
| 15 |
+
|
| 16 |
+
Based on the results returned by the tool(s), you can call additional tools if needed, correct tool calls if any errors are found, or just respond with the answer to the user.' -}}{%- endif -%}{{- eot -}}{%- for message in messages -%}{%- if message.role == user_token -%}{{- start_header ~ user_token ~ end_header -}}{{ message.content -}}{{ eot -}}{%- elif message.role == assistant_token -%}{%- if '</think>' in message.content -%}{%- set content = (message.content.split('</think>')|last).lstrip() -%}{%- else -%}{%- set content = message.content -%}{%- endif -%}{{- start_header ~ assistant_token ~ end_header -}}{{ content -}}{%- if message.tool_calls -%}{{- '<TOOLCALL>[' -}}{%- for call in message.tool_calls -%}{%- set fn = call.function if call.function is defined else call -%}{{- '{"name": "' + fn.name + '", "arguments": ' -}}{%- if fn.arguments is string -%}{{- fn.arguments -}}{%- else -%}{{- fn.arguments | tojson -}}{%- endif -%}{{- '}' + (', ' if not loop.last else '') -}}{%- endfor -%}{{- ']</TOOLCALL>' -}}{%- endif -%}{{- eot -}}{%- elif message.role == tool_token -%}{%- if loop.first or (messages[loop.index0 - 1].role != tool_token) -%}{{- start_header ~ tool_token ~ end_header -}}{{ '<TOOL_RESPONSE>[' -}}{%- endif -%}{{- message.content -}}{{- ', ' if not loop.last and (messages[loop.index0 + 1].role == tool_token) else '' -}}{%- if loop.last or (messages[loop.index0 + 1].role != tool_token) -%}{{- ']</TOOL_RESPONSE>' -}}{{ eot -}}{%- endif -%}{%- endif -%}{%- endfor -%}{%- if add_generation_prompt -%}{{- start_header ~ assistant_token ~ end_header -}}{%- if not enable_thinking -%}{{- '<think>
|
| 17 |
+
|
| 18 |
+
</think>
|
| 19 |
+
|
| 20 |
+
' -}}{%- endif -%}{%- endif -%}
|