Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,15 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text
+accuracy_plot.png filter=lfs diff=lfs merge=lfs -text
+agentic_coding_benchmarks.png filter=lfs diff=lfs merge=lfs -text
+nemotron-3.5-lightning-30b-a3b-Q4_0.gguf filter=lfs diff=lfs merge=lfs -text
+nemotron-3.5-lightning-30b-a3b-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+nemotron-3.5-lightning-30b-a3b-Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+nemotron-3.5-lightning-30b-a3b-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+nemotron-3.5-lightning-30b-a3b-Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+nemotron-3.5-lightning-30b-a3b-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
+nemotron-3.5-lightning-30b-a3b-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
+nemotron-3.5-lightning-30b-a3b-bf16.gguf filter=lfs diff=lfs merge=lfs -text
+nemotron-3.5-lightning-30b-a3b-dflash.gguf filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,194 @@
+---
+base_model:
+- nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16
+language:
+- en
+- es
+- fr
+- de
+- it
+- ja
+library_name: transformers
+license: openmdw-1.1
+license_link: https://openmdw.ai/license/1-1/
+pipeline_tag: text-generation
+tags:
+- nvidia
+- nemotron-3.5
+- gguf
+- llama.cpp
+- text-generation
+- moe
+---
+
+# NVIDIA-Nemotron-3.5-Lightning-30B-A3B - GGUF
+
+This repository contains GGUF format model files for [NVIDIA's NVIDIA-Nemotron-3.5-Lightning-30B-A3B](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16).
+
+These files were converted and quantized using [llama.cpp](https://github.com/ggerganov/llama.cpp).
+
+## Available Files
+
+| Filename | Quant Method | Description |
+| --- | --- | --- |
+| `nemotron-3.5-lightning-30b-a3b-bf16.gguf` | BF16 | Full-precision reference weights (unquantized) |
+| `nemotron-3.5-lightning-30b-a3b-Q8_0.gguf` | Q8_0 | Extremely high quality, fast, high resource usage |
+| `nemotron-3.5-lightning-30b-a3b-Q6_K.gguf` | Q6_K | Very high quality, near-lossless quantization |
+| `nemotron-3.5-lightning-30b-a3b-Q5_K_M.gguf` | Q5_K_M | High quality, balanced performance and memory |
+| `nemotron-3.5-lightning-30b-a3b-Q5_K_S.gguf` | Q5_K_S | High quality, slightly smaller footprint than Q5_K_M |
+| `nemotron-3.5-lightning-30b-a3b-Q4_K_M.gguf` | Q4_K_M | Recommended balance of size, speed, and quality |
+| `nemotron-3.5-lightning-30b-a3b-Q4_K_S.gguf` | Q4_K_S | 4-bit quantization with small memory footprint |
+| `nemotron-3.5-lightning-30b-a3b-Q4_0.gguf` | Q4_0 | Standard 4-bit quantization |
+| `nemotron-3.5-lightning-30b-a3b-dflash.gguf` | — | DFlash speculative decoding draft model |
+
+## Model Summary
+
+| Total Parameters | 30B (3B active) |
+| --- | --- |
+| Architecture | MoE — Mamba-2 + MoE + Attention hybrid |
+| Context Length | Up to 1M tokens (256K native default) |
+| Supported Languages | English (and coding languages), Spanish, French, German, Italian, Japanese |
+| Speculative Decoding | DSpark, DFlash, MTP (Multi-Token Prediction) |
+| Reasoning Mode | Configurable on/off via chat template (`enable_thinking=True/False`) |
+| Recommended Sampling | Temperature 1.0, Top_P 0.95 |
+| License | OpenMDW License Agreement, version 1.1 |
+| Original Model | [nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16) |
+
+## Model Overview
+
+Model Developer: NVIDIA Corporation
+
+Model Dates: December 2025 - May 2026
+
+Data Freshness:
+- The pre-training data has a cutoff date of September 2025.
+- The post-training data has a cutoff date of May 2026.
+
+### What is Nemotron?
+
+NVIDIA Nemotron™ is a family of open models with open weights, training data, and recipes, delivering leading efficiency and accuracy for building specialized AI agents.
+
+## Description
+
+NVIDIA-Nemotron-3.5-Lightning-30B-A3B is a large language model (LLM) trained by NVIDIA.
+
+The model employs a hybrid Mixture-of-Experts architecture, utilizing interleaved Mamba-2 and MoE layers, along with select Attention layers. The Lightning 3.5 model is released alongside speculative decoding methods (DSpark, DFlash, MTP) for faster text generation. The model has 3B active parameters and 30B parameters in total.
+
+This model is ready for commercial use under the OpenMDW-1.1 license.
+
+## Usage with llama.cpp
+
+### CLI / llama-cli
+
+Reasoning ON (default):
+
+```bash
+llama-cli \
+  -m nemotron-3.5-lightning-30b-a3b-Q4_K_M.gguf \
+  --jinja \
+  --chat-template-file chat_template.jinja \
+  -p "Write a Python function to compute Fibonacci numbers." \
+  --temp 1.0 --top-p 0.95 \
+  -ngl 99
+```
+
+### llama-server
+
+Start the OpenAI-compatible server:
+
+```bash
+llama-server \
+  -m nemotron-3.5-lightning-30b-a3b-Q4_K_M.gguf \
+  --temp 1.0 --top-p 0.95 \
+  -np 1 \
+  -c 40960 \
+  --port 8000 \
+  -ngl 99 \
+  -fa on \
+  --jinja \
+  --chat-template-file chat_template.jinja \
+  --no-webui \
+  --fit off
+```
+
+#### With DFlash Speculative Decoding
+
+Accelerate token generation using the DFlash draft model:
+
+```bash
+llama-server \
+  -m nemotron-3.5-lightning-30b-a3b-Q4_K_M.gguf \
+  -md nemotron-3.5-lightning-30b-a3b-dflash.gguf \
+  --draft-max 6 \
+  --temp 1.0 --top-p 0.95 \
+  -np 1 \
+  -c 40960 \
+  --port 8000 \
+  -ngl 99 \
+  -ngld 99 \
+  -fa on \
+  --jinja \
+  --chat-template-file chat_template.jinja \
+  --no-webui \
+  --fit off
+```
+
+### API Client Example (OpenAI SDK)
+
+```python
+from openai import OpenAI
+
+client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
+
+# Reasoning ON (default)
+response = client.chat.completions.create(
+    model="nemotron-3.5-lightning-30b-a3b",
+    messages=[{"role": "user", "content": "Write a haiku about GPUs"}],
+    max_tokens=4096,
+    temperature=1.0,
+    top_p=0.95,
+    extra_body={"chat_template_kwargs": {"enable_thinking": True}},
+)
+print(response.choices[0].message.content)
+
+# Reasoning OFF (direct answer)
+response = client.chat.completions.create(
+    model="nemotron-3.5-lightning-30b-a3b",
+    messages=[{"role": "user", "content": "What is the capital of Japan?"}],
+    max_tokens=128,
+    temperature=1.0,
+    top_p=0.95,
+    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
+)
+print(response.choices[0].message.content)
+```
+
+## Benchmarks
+
+### Reasoning Benchmark Evaluations
+
+| Task | Nemotron-3.5-Lightning-30B-A3B-BF16 | Nemotron-3.5-Lightning-30B-A3B-NVFP4 |
+| --- | --- | --- |
+| **General Knowledge** | | |
+| MMLU Pro | 81.94 | 81.62 |
+| AA-Omniscience | 17.50 | 16.63 |
+| **Reasoning** | | |
+| GPQA Diamond (no tools) | 75.44 | 75.57 |
+| HLE (text-only, no tools) | 11.72 | 10.47 |
+| SciCode | 32.60 | 31.38 |
+| **Coding & Agentic** | | |
+| SWE-bench Verified | 51.56 | 52.80 |
+| SWE-bench Multilingual | 39.33 | 36.47 |
+| Terminal-Bench 2.1 | 24.58 | 23.46 |
+| PinchBench | 85.37 | 83.43 |
+| BrowseComp | 36.97 | 36.81 |
+| τ³-bench (Banking) | 9.28 | 9.48 |
+| GDPval-AA-V2 | 832 | 865 |
+| **Instruction Following** | | |
+| IFBench (loose) | 71.88 | 72.88 |
+| **Long Context** | | |
+| AA-LCR | 52.00 | 49.19 |
+
+## License and Terms of Use
+
+Governing Download Terms: Use of this model is governed by the OpenMDW-1.1 model license.

chat_template.jinja

@@ -0,0 +1,190 @@
+{% macro render_extra_keys(json_dict, handled_keys) %}
+    {%- if json_dict is mapping %}
+        {%- for json_key in json_dict if json_key not in handled_keys %}
+            {%- if json_dict[json_key] is mapping or (json_dict[json_key] is sequence and json_dict[json_key] is not string) %}
+                {{- '\n<' ~ json_key ~ '>' ~ (json_dict[json_key] | tojson | safe) ~ '</' ~ json_key ~ '>' }}
+            {%- else %}
+                {{-'\n<' ~ json_key ~ '>' ~ (json_dict[json_key] | string) ~ '</' ~ json_key ~ '>' }}
+            {%- endif %}
+        {%- endfor %}
+    {%- endif %}
+{% endmacro %}
+{%- set enable_thinking = enable_thinking if enable_thinking is defined else True %}
+{%- set truncate_history_thinking = truncate_history_thinking if truncate_history_thinking is defined else True %}
+{%- set ns = namespace(last_user_idx = -1) %}
+{%- set loop_messages = messages %}
+{%- for m in loop_messages %}
+  {%- if m["role"] == "user" %}
+    {%- set ns.last_user_idx = loop.index0 %}
+  {%- endif %}
+{%- endfor %}
+{%- if messages[0]["role"] == "system" %}
+    {%- set system_message = messages[0]["content"] %}
+    {%- set loop_messages = messages[1:] %}
+{%- else %}
+    {%- set system_message = "" %}
+    {%- set loop_messages = messages %}
+{%- endif %}
+{%- if not tools is defined %}
+    {%- set tools = [] %}
+{%- endif %}
+{%- set ns = namespace(last_user_idx = -1) %}
+{%- for m in loop_messages %}
+  {%- if m["role"] == "user" %}
+    {%- set ns.last_user_idx = loop.index0 %}
+  {%- endif %}
+{%- endfor %}
+{%- if system_message is defined %}
+    {{- "<|im_start|>system\n" + system_message }}
+{%- else %}
+    {%- if tools is iterable and tools | length > 0 %}
+        {{- "<|im_start|>system\n" }}
+    {%- endif %}
+{%- endif %}
+{%- if tools is iterable and tools | length > 0 %}
+    {%- if system_message is defined and system_message | length > 0 %}
+        {{- "\n\n" }}
+    {%- endif %}
+    {{- "# Tools\n\nYou have access to the following functions:\n\n" }}
+    {{- "<tools>" }}
+    {%- for tool in tools %}
+        {%- if tool.function is defined %}
+            {%- set tool = tool.function %}
+        {%- endif %}
+        {{- "\n<function>\n<name>" ~ tool.name ~ "</name>" }}
+        {%- if tool.description is defined %}
+            {{- '\n<description>' ~ (tool.description | trim) ~ '</description>' }}
+        {%- endif %}
+        {{- '\n<parameters>' }}
+        {%- if tool.parameters is defined and tool.parameters is mapping and tool.parameters.properties is defined and tool.parameters.properties is mapping %}
+            {%- for param_name, param_fields in tool.parameters.properties|items %}
+                {{- '\n<parameter>' }}
+                {{- '\n<name>' ~ param_name ~ '</name>' }}
+                {%- if param_fields.type is defined %}
+                    {{- '\n<type>' ~ (param_fields.type | string) ~ '</type>' }}
+                {%- endif %}
+                {%- if param_fields.description is defined %}
+                    {{- '\n<description>' ~ (param_fields.description | trim) ~ '</description>' }}
+                {%- endif %}
+                {%- if param_fields.enum is defined %}
+                    {{- '\n<enum>' ~ (param_fields.enum | tojson | safe) ~ '</enum>' }}
+                {%- endif %}
+                {%- set handled_keys = ['name', 'type', 'description', 'enum'] %}
+                {{- render_extra_keys(param_fields, handled_keys) }}
+                {{- '\n</parameter>' }}
+            {%- endfor %}
+        {%- endif %}
+        {% set handled_keys = ['type', 'properties', 'required'] %}
+        {{- render_extra_keys(tool.parameters, handled_keys) }}
+        {%- if tool.parameters is defined and tool.parameters.required is defined %}
+            {{- '\n<required>' ~ (tool.parameters.required | tojson | safe) ~ '</required>' }}
+        {%- endif %}
+        {{- '\n</parameters>' }}
+        {%- set handled_keys = ['type', 'name', 'description', 'parameters'] %}
+        {{- render_extra_keys(tool, handled_keys) }}
+        {{- '\n</function>' }}
+    {%- endfor %}
+    {{- "\n</tools>" }}
+    {{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n<tool_call>\n<function=example_function_name>\n<parameter=example_parameter_1>\nvalue_1\n</parameter>\n<parameter=example_parameter_2>\nThis is the value for the second parameter\nthat can span\nmultiple lines\n</parameter>\n</function>\n</tool_call>\n\n<IMPORTANT>\nReminder:\n- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags\n- Required parameters MUST be specified\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\n</IMPORTANT>' }}
+{%- endif %}
+{%- if system_message is defined %}
+    {{- '<|im_end|>\n' }}
+{%- else %}
+    {%- if tools is iterable and tools | length > 0 %}
+        {{- '<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- for message in loop_messages %}
+    {%- if message.role == "assistant" %}
+        {%- if message.reasoning_content is defined and message.reasoning_content is string and message.reasoning_content | trim | length > 0 %}
+            {%- set content = "<think>\n" ~ message.reasoning_content ~ "</think>" ~ (message.content | default('', true)) %}
+        {%- else %}
+            {%- set content = message.content | default('', true) %}
+            {%- if content is string -%}
+                {%- if '<think>' not in content and '</think>' not in content -%}
+                    {%- set content = "<think></think>" ~ content -%}
+                {%- endif -%}
+            {%- else -%}
+                {%- set content = content -%}
+            {%- endif -%}
+        {%- endif %}
+        {%- if message.tool_calls is defined and message.tool_calls is iterable and message.tool_calls | length > 0 %}
+            {{- '<|im_start|>assistant\n' }}
+                {%- set include_content = not (truncate_history_thinking and loop.index0 < ns.last_user_idx) %}
+                {%- if content is string and content | trim | length > 0 %}
+                    {%- if include_content %}
+                        {{- (content | trim) ~ '\n' -}}
+                    {%- else %}
+                        {%- set c = (content | string) %}
+                        {%- if '</think>' in c %}
+                            {%- set c = c.split('</think>')[-1] %}
+                        {%- elif '<think>' in c %}
+                            {%- set c = c.split('<think>')[0] %}
+                        {%- endif %}
+                        {%- set c = "<think></think>" ~ c %}
+                        {%- if c | length > 0 %}
+                            {{- c ~ '\n' -}}
+                        {%- endif %}
+                    {%- endif %}
+                {%- else %}
+                    {{- "<think></think>" -}}
+                {%- endif %}
+                {%- for tool_call in message.tool_calls %}
+                    {%- if tool_call.function is defined %}
+                        {%- set tool_call = tool_call.function %}
+                    {%- endif %}
+                    {{- '<tool_call>\n<function=' ~ tool_call.name ~ '>\n' -}}
+                        {%- if tool_call.arguments is defined %}
+                            {%- for args_name, args_value in tool_call.arguments|items %}
+                                {{- '<parameter=' ~ args_name ~ '>\n' -}}
+                                    {%- set args_value = args_value | tojson | safe if args_value is mapping or (args_value is sequence and args_value is not string) else args_value | string %}
+                                {{- args_value ~ '\n</parameter>\n' -}}
+                            {%- endfor %}
+                        {%- endif %}
+                    {{- '</function>\n</tool_call>\n' -}}
+                {%- endfor %}
+                {{- '<|im_end|>\n' }}
+        {%- else %}
+            {%- if not (truncate_history_thinking and loop.index0 < ns.last_user_idx) %}
+                {{- '<|im_start|>assistant\n' ~ (content | default('', true) | string | trim) ~ '<|im_end|>\n' }}
+            {%- else %}
+                {%- set c = (content | default('', true) | string) %}
+                {%- if '<think>' in c and '</think>' in c %}
+                    {%- set c = "<think></think>" ~ c.split('</think>')[-1] %}
+                {%- endif %}
+                {%- set c = c | trim %}
+                {%- if c | length > 0 %}
+                    {{- '<|im_start|>assistant\n' ~ c ~ '<|im_end|>\n' }}
+                {%- else %}
+                    {{- '<|im_start|>assistant\n<|im_end|>\n' }}
+                {%- endif %}
+            {%- endif %}
+        {%- endif %}
+    {%- elif message.role == "user" or message.role == "system" %}
+        {{- '<|im_start|>' + message.role + '\n' }}
+        {%- set content = message.content | string %}
+        {{- content }}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if loop.previtem and loop.previtem.role != "tool" %}
+            {{- '<|im_start|>user\n' }}
+        {%- endif %}
+        {{- '<tool_response>\n' }}
+        {{- message.content }}
+        {{- '\n</tool_response>\n' }}
+        {%- if not loop.last and loop.nextitem.role != "tool" %}
+            {{- '<|im_end|>\n' }}
+        {%- elif loop.last %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- else %}
+        {{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>\n' }}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {%- if enable_thinking %}
+        {{- '<|im_start|>assistant\n<think>\n' }}
+    {%- else %}
+        {{- '<|im_start|>assistant\n<think></think>' }}
+    {%- endif %}
+{%- endif %}

nemotron-3.5-lightning-30b-a3b-Q4_0.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ef78f377618e020d3bb50fc831fd0edc113b468aea87a9996aa99e683c3a3a0c
+size 18728781472

nemotron-3.5-lightning-30b-a3b-Q4_K_M.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:50d2ce39590c44b2db38531faf03eb9956658d58b8e778add38eb8f74d67959b
+size 25430739616

nemotron-3.5-lightning-30b-a3b-Q4_K_S.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:12294213e4a9001140c75f01be7ead4ebcfba443bfd99707c6ddcb7d5414c181
+size 22835894944

nemotron-3.5-lightning-30b-a3b-Q5_K_M.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:920fa53845bca15e0461244c39b86d09c8d5eb639e7e58abfbeba1c6bf739caa
+size 27040754848

nemotron-3.5-lightning-30b-a3b-Q5_K_S.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0cfd27ccf850ee7ad8534b7eec46ec2b13b9f11c628a9965ab1b699232512506
+size 24810682528

nemotron-3.5-lightning-30b-a3b-Q6_K.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:56bcf2d5062f69593a42da794714da9a29a7c7dd3f7b78d8ed1cca1f4a192738
+size 34921148320

nemotron-3.5-lightning-30b-a3b-Q8_0.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:de89886ff4c2c9c8dcc9049bff00732f2b393f2af4161b69ec4a0fefbedeafbf
+size 35004642976

nemotron-3.5-lightning-30b-a3b-bf16.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7e0330212140b1c6f9b20bcb208b26b4cccdcfb51dc2c3bf1b5afb7663ed928e
+size 65852184736

nemotron-3.5-lightning-30b-a3b-dflash.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1d8344244f363ec678ce17eb93443761cbb9226779a76bf162243fc3f73f9f8a
+size 1185032704