Add Mac memory chooser and reproducible demo prompt

README.md

@@ -26,18 +26,22 @@ tags:
   - 4-bit
 ---
 
+
 <p align="center">
   <img src="https://img.shields.io/badge/NVIDIA-Nemotron-76B900?style=for-the-badge&logo=nvidia&logoColor=white" alt="NVIDIA Nemotron">
   <img src="https://img.shields.io/badge/Apple_Silicon-MLX-000000?style=for-the-badge&logo=apple&logoColor=white" alt="Apple silicon MLX">
   <img src="https://img.shields.io/badge/Vontra-oMLX-6E56CF?style=for-the-badge&logo=huggingface&logoColor=white" alt="Vontra oMLX">
 </p>
 
+
 <h1 align="center">NVIDIA Nemotron 3.5 Lightning 30B-A3B — MLX 4-bit</h1>
 
+
 <p align="center">
   A native Apple-silicon conversion of <a href="https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16">nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16</a>, quantized with stock 4-bit affine weights and packaged for MLX-LM and oMLX.
 </p>
 
+
 <p align="center">
   <a href="https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16">Original model</a> ·
   <a href="https://developer.nvidia.com/nemotron">NVIDIA Nemotron</a> ·
@@ -45,10 +49,13 @@ tags:
   <a href="https://openmdw.ai/license/1-1/">OpenMDW 1.1 license</a>
 </p>
 
+
 ## About this conversion
 
+
 This repository contains a stock 4-bit affine MLX conversion of NVIDIA Nemotron 3.5 Lightning. The source is a 30B-total / 3B-active hybrid mixture-of-experts model that interleaves Mamba-2, sparse MoE, and attention layers. The upstream tokenizer, chat template, and generation configuration are preserved.
 
+
 | Item | Value |
 | --- | --- |
 | Base model | [`nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16`](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16) |
@@ -60,12 +67,16 @@ This repository contains a stock 4-bit affine MLX conversion of NVIDIA Nemotron
 | Maximum configured context | 262,144 tokens |
 | Architecture | `nemotron_h` — Mamba-2 + sparse MoE + attention |
 
+
 > [!NOTE] MLX-LM reported an effective precision of 4.503 bits per weight.
 
+
 ## Apple-silicon performance
 
+
 This checkpoint was load-tested and generation-tested on the following machine:
 
+
 | Hardware | Configuration |
 | --- | --- |
 | Host | Mac Studio |
@@ -74,8 +85,10 @@ This checkpoint was load-tested and generation-tested on the following machine:
 | Unified memory | 256 GB |
 | Runtime | MLX-LM 0.31.3 / MLX 0.32.0 |
 
+
 A warmed local test produced:
 
+
 | Measurement | Result |
 | --- | ---: |
 | Decode (median) | **168.41 tokens/s** |
@@ -84,18 +97,24 @@ A warmed local test produced:
 | Warm-up | 32 generated tokens |
 | Prompt | 36 tokens after chat templating |
 
+
 The decode figure is the median of three greedy 256-token runs after a 32-token Metal-kernel warm-up. It is a practical local reference, not a controlled cross-platform benchmark. Prompt length, context growth, sampler settings, memory pressure, thermal state, and MLX/oMLX versions can materially change performance.
 
+
 ## Quick start with MLX-LM
 
+
 Install recent MLX-LM and Hugging Face tooling:
 
+
 ```bash
 python -m pip install -U mlx-lm huggingface_hub
 ```
 
+
 Run directly from the Hub:
 
+
 ```bash
 mlx_lm.generate \
   --model Vontra/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit \
@@ -105,8 +124,10 @@ mlx_lm.generate \
   --top-p 0.95
 ```
 
+
 Reasoning mode is enabled by the upstream chat template by default. To disable it:
 
+
 ```bash
 mlx_lm.generate \
   --model Vontra/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit \
@@ -115,11 +136,14 @@ mlx_lm.generate \
   --max-tokens 256
 ```
 
+
 Python usage:
 
+
 ```python
 from mlx_lm import load, generate
 
+
 model, tokenizer = load("Vontra/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit")
 messages = [
     {"role": "user", "content": "Explain sparse mixture-of-experts routing."}
@@ -133,21 +157,27 @@ prompt = tokenizer.apply_chat_template(
 print(generate(model, tokenizer, prompt=prompt, max_tokens=512))
 ```
 
+
 To download the repository first:
 
+
 ```bash
 hf download Vontra/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit \
   --local-dir ~/.omlx/models/Vontra/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit
 ```
 
+
 ## Using it with oMLX
 
+
 1. Place the downloaded model at `~/.omlx/models/Vontra/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit`.
 2. Refresh the oMLX model registry.
 3. Load `NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit` and use the normal chat UI or OpenAI-compatible endpoint.
 
+
 Example request:
 
+
 ```bash
 curl http://localhost:8000/v1/chat/completions \
   -H "Content-Type: application/json" \
@@ -161,12 +191,16 @@ curl http://localhost:8000/v1/chat/completions \
   }'
 ```
 
+
 For long prompts, begin with a conservative context limit and increase it while watching memory pressure. The configured 256K context is a model capability, not a guarantee that every host can prefill that context within its available unified memory.
 
+
 ## Architecture
 
+
 Nemotron 3.5 Lightning is a hybrid sparse model designed for efficient agentic and reasoning workloads.
 
+
 | Architecture detail | Upstream value |
 | --- | ---: |
 | Total / active parameters | 30B / 3B |
@@ -179,10 +213,13 @@ Nemotron 3.5 Lightning is a hybrid sparse model designed for efficient agentic a
 | Vocabulary size | 131,072 |
 | Configured context | 262,144 tokens |
 
+
 The upstream release is intended for coding, tool use, reasoning, research, and customization. For NVIDIA's evaluations, deployment guidance, intended use, limitations, safety information, and full architecture discussion, see the [original model card](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16).
 
+
 ## Conversion and validation notes
 
+
 - Source weights: NVIDIA's BF16 checkpoint.
 - Quantization group size: 64.
 - Quantization mode: affine.
@@ -191,10 +228,46 @@ The upstream release is intended for coding, tool use, reasoning, research, and
 - The model was loaded and exercised through end-to-end generation on Apple silicon.
 - Quantization can reduce output quality relative to BF16; use a higher-precision variant when quality matters more than memory use.
 
+
 This is a community conversion, not an official NVIDIA release. Validate quality and numerical behavior on your own representative workload before production use.
 
+
 ## License and attribution
 
+
 The upstream model is released under the **OpenMDW License Agreement, version 1.1**. A copy is included in this repository; review it before use or redistribution.
 
+
 All model design, training, benchmark, and upstream documentation credit belongs to NVIDIA and the original contributors. The MLX conversion, Apple-silicon validation, and packaging are provided by [Vontra](https://huggingface.co/Vontra).
+
+
+
+<!-- vontra-chooser-start -->
+## Choose for your Mac
+
+[64GB Macs](https://huggingface.co/collections/Vontra/mlx-models-for-64gb-macs-6a9fefda17932216ec9ab457) · [128GB Macs](https://huggingface.co/collections/Vontra/mlx-models-for-128gb-macs-6a9ff0abd31bc9abbe7922d7) · [256GB Macs](https://huggingface.co/collections/Vontra/mlx-models-for-256gb-macs-6a9ff0ef9fed7c5bdca15e9b)
+
+Published peak memory: **17.95 GB**; estimated starting tier: **64GB**, leaving about **46 GB** nominal headroom. The collections use published M3 Studio peaks with at least 25% nominal headroom; fit on other Macs is an estimate, and full context is not guaranteed. Start with short context and one request.
+
+### Runtime and evidence
+
+The exact tested oMLX application version is not recorded here; a library version is not an app version. The original performance tables retain their benchmark conditions and speed figures; this documentation update adds no new test results.
+
+### Quick start and demo prompt
+
+```bash
+hf download Vontra/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit --local-dir ./models/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit
+```
+
+Add the downloaded folder to oMLX model directories, refresh the list, and follow this card's architecture and MTP compatibility requirements before loading.
+
+Try this in a new chat with a 128-token output limit:
+
+```text
+Explain why the sky looks blue in three short sentences.
+```
+
+This is a demo prompt to try, not a recorded successful run; a captured demonstration for this documentation update is not yet available.
+
+[Follow Vontra for new Apple Silicon releases and fixes.](https://huggingface.co/Vontra)
+<!-- vontra-chooser-end -->