Update README.md
Browse files
README.md
CHANGED
|
@@ -20,7 +20,7 @@ tags:
|
|
| 20 |
# Qwopus3.6-35B-A3B-v1-oQ8-MTP
|
| 21 |
Unofficial **oQ8 MLX/oMLX quantized conversion** of [`Jackrong/Qwopus3.6-35B-A3B-v1`](https://huggingface.co/Jackrong/Qwopus3.6-35B-A3B-v1).
|
| 22 |
This model was quantized using [oQ / oMLX](https://github.com/jundot/omlx) mixed-precision quantization with **oMLX v0.3.9**.
|
| 23 |
-
This conversion is intended for **oMLX / MLX inference on Apple Silicon**
|
| 24 |
> This is not an official Qwen, Alibaba, or QwOpus release.
|
| 25 |
> It is an unofficial community quantized conversion for local inference.
|
| 26 |
## Redownload notice
|
|
@@ -34,7 +34,6 @@ This conversion includes **oQ8 weights** and **native MTP-related files/configur
|
|
| 34 |
- Sparse MoE model
|
| 35 |
- 35B total parameters / approximately 3B active parameters per token
|
| 36 |
- Native long-context support
|
| 37 |
-
- Designed for reasoning, coding, agentic workflows, tool use, and long-context tasks
|
| 38 |
## Quantization details
|
| 39 |
- **Model type**: `qwen3_5_moe`
|
| 40 |
- **Quantization method**: oQ mixed-precision quantization
|
|
@@ -44,14 +43,14 @@ This conversion includes **oQ8 weights** and **native MTP-related files/configur
|
|
| 44 |
- **Runtime used for conversion**: oMLX v0.3.9
|
| 45 |
- **Native MTP**: preserved / intended to be used
|
| 46 |
- **Additional fine-tuning**: none
|
| 47 |
-
##
|
| 48 |
Tested primarily with:
|
| 49 |
- **Runtime**: oMLX v0.3.9
|
| 50 |
- **Hardware**: Apple Silicon, tested on Mac Studio M4 Max 128GB
|
| 51 |
- **Context tested**: up to 262144 ctx
|
| 52 |
- **MTP**: ON
|
| 53 |
- **DFlash**: OFF
|
| 54 |
-
Recommended initial
|
| 55 |
```text
|
| 56 |
MTP: ON
|
| 57 |
DFlash: OFF
|
|
@@ -73,6 +72,12 @@ MTP ON low-context peak ~70 tok/s
|
|
| 73 |
|
| 74 |
These are local observations, not standardized benchmarks. Actual speed depends on prompt length, context length, oMLX version, generation settings, temperature, cache behavior, and Apple Silicon memory pressure.
|
| 75 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 76 |
Known notes
|
| 77 |
|
| 78 |
* Native MTP showed a measurable decode-speed improvement in local testing.
|
|
@@ -100,4 +105,4 @@ Compared with the base model, this repository:
|
|
| 100 |
* converts the original model weights to oQ8 MLX safetensors
|
| 101 |
* preserves native MTP-related files/configuration
|
| 102 |
* does not add additional fine-tuning
|
| 103 |
-
* is
|
|
|
|
| 20 |
# Qwopus3.6-35B-A3B-v1-oQ8-MTP
|
| 21 |
Unofficial **oQ8 MLX/oMLX quantized conversion** of [`Jackrong/Qwopus3.6-35B-A3B-v1`](https://huggingface.co/Jackrong/Qwopus3.6-35B-A3B-v1).
|
| 22 |
This model was quantized using [oQ / oMLX](https://github.com/jundot/omlx) mixed-precision quantization with **oMLX v0.3.9**.
|
| 23 |
+
This conversion preserves the model's **native MTP** support and is intended for **oMLX / MLX inference on Apple Silicon**.
|
| 24 |
> This is not an official Qwen, Alibaba, or QwOpus release.
|
| 25 |
> It is an unofficial community quantized conversion for local inference.
|
| 26 |
## Redownload notice
|
|
|
|
| 34 |
- Sparse MoE model
|
| 35 |
- 35B total parameters / approximately 3B active parameters per token
|
| 36 |
- Native long-context support
|
|
|
|
| 37 |
## Quantization details
|
| 38 |
- **Model type**: `qwen3_5_moe`
|
| 39 |
- **Quantization method**: oQ mixed-precision quantization
|
|
|
|
| 43 |
- **Runtime used for conversion**: oMLX v0.3.9
|
| 44 |
- **Native MTP**: preserved / intended to be used
|
| 45 |
- **Additional fine-tuning**: none
|
| 46 |
+
## Runtime notes
|
| 47 |
Tested primarily with:
|
| 48 |
- **Runtime**: oMLX v0.3.9
|
| 49 |
- **Hardware**: Apple Silicon, tested on Mac Studio M4 Max 128GB
|
| 50 |
- **Context tested**: up to 262144 ctx
|
| 51 |
- **MTP**: ON
|
| 52 |
- **DFlash**: OFF
|
| 53 |
+
Recommended initial runtime flags based on local testing:
|
| 54 |
```text
|
| 55 |
MTP: ON
|
| 56 |
DFlash: OFF
|
|
|
|
| 72 |
|
| 73 |
These are local observations, not standardized benchmarks. Actual speed depends on prompt length, context length, oMLX version, generation settings, temperature, cache behavior, and Apple Silicon memory pressure.
|
| 74 |
|
| 75 |
+
Format notes
|
| 76 |
+
|
| 77 |
+
This is not GGUF, AWQ, GPTQ, or bitsandbytes format.
|
| 78 |
+
|
| 79 |
+
This repository contains an MLX safetensors conversion intended for oMLX / MLX-compatible runtimes.
|
| 80 |
+
|
| 81 |
Known notes
|
| 82 |
|
| 83 |
* Native MTP showed a measurable decode-speed improvement in local testing.
|
|
|
|
| 105 |
* converts the original model weights to oQ8 MLX safetensors
|
| 106 |
* preserves native MTP-related files/configuration
|
| 107 |
* does not add additional fine-tuning
|
| 108 |
+
* is packaged for local oMLX / Apple Silicon usage
|