Add MNN Q4 conversion for TokForge mobile inference

.gitattributes

@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+llm.mnn filter=lfs diff=lfs merge=lfs -text
+llm.mnn.weight filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,99 @@
+---
+license: apache-2.0
+language:
+  - en
+pipeline_tag: text-generation
+base_model: cognitivecomputations/Dolphin-Mistral-24B-Venice-Edition
+tags:
+  - mnn
+  - mistral
+  - mobile
+  - on-device
+  - tokforge
+  - uncensored
+  - abliterated
+---
+
+# Dolphin-Mistral-24B-Venice-Edition-MNN
+
+Pre-converted [Dolphin-Mistral-24B-Venice-Edition](https://huggingface.co/cognitivecomputations/Dolphin-Mistral-24B-Venice-Edition) in MNN format for on-device inference with [TokForge](https://tokforge.ai).
+
+> **Original model by [cognitivecomputations](https://huggingface.co/cognitivecomputations)** — converted to MNN Q4 for mobile deployment.
+
+## Model Details
+
+| | |
+|---|---|
+| **Architecture** | Mistral Small 24B (standard attention, 40 layers) |
+| **Parameters** | 24B (4-bit quantized) |
+| **Format** | MNN (Alibaba Mobile Neural Network) |
+| **Quantization** | W4A16 (4-bit weights, block size 128) |
+| **Vocab** | 32,768 tokens |
+| **Source** | [cognitivecomputations/Dolphin-Mistral-24B-Venice-Edition](https://huggingface.co/cognitivecomputations/Dolphin-Mistral-24B-Venice-Edition) |
+
+## Description
+
+Dolphin Mistral 24B Venice Edition — Venice AI's most uncensored model, developed in collaboration with Eric Hartford's Dolphin team. The largest and most capable uncensored model in this collection. Requires 24GB+ RAM (flagship phones with 24GB only).
+
+## Files
+
+| File | Description |
+|------|-------------|
+| `llm.mnn` | Model computation graph |
+| `llm.mnn.weight` | Quantized weight data (Q4, block=128) |
+| `llm_config.json` | Model config with Jinja chat template |
+| `tokenizer.txt` | Tokenizer vocabulary |
+| `config.json` | MNN runtime config |
+
+## Usage with TokForge
+
+This model is optimized for **[TokForge](https://tokforge.ai)** — a free Android app for private, on-device LLM inference.
+
+1. Download [TokForge from the Play Store](https://tokforge.ai)
+2. Open the app → Models → Download this model
+3. Start chatting — runs 100% locally, no internet required
+
+### Recommended Settings
+
+| Setting | Value |
+|---------|-------|
+| Backend | OpenCL (Qualcomm) / Vulkan (MediaTek) / CPU (fallback) |
+| Precision | Low |
+| Threads | 4 |
+| Thinking | Off (or On for thinking-capable models) |
+
+
+
+## Performance
+
+Actual speed varies by device, thermal state, and generation length. Typical ranges for this model size:
+
+| Device | SoC | Backend | tok/s |
+|---|---|---|---|
+| RedMagic 11 Pro (24GB) | SM8850 | OpenCL | **5.4 tok/s** |
+
+> **Note:** Requires 24GB+ RAM. May not sustain long conversations on 24GB devices due to KV cache memory pressure. Best on tablets or phones with 24GB+ RAM and minimal background apps.
+
+## Attribution
+
+This is an MNN conversion of **[Dolphin-Mistral-24B-Venice-Edition](https://huggingface.co/cognitivecomputations/Dolphin-Mistral-24B-Venice-Edition)** by **[cognitivecomputations](https://huggingface.co/cognitivecomputations)**. All credit for the model architecture, training, and fine-tuning goes to the original author(s). This conversion only changes the runtime format for mobile deployment.
+
+## Limitations
+
+- Intended for TokForge / MNN on-device inference on Android
+- This is a runtime bundle, not a standard Transformers training checkpoint
+- Quantization (Q4) may slightly reduce quality compared to the full-precision original
+- Abliterated/uncensored models have had safety filters removed — **use responsibly**
+
+## Community
+
+- **Website:** [tokforge.ai](https://tokforge.ai)
+- **Discord:** [Join our Discord](https://discord.gg/Acv3CBtfVm)
+- **GitHub:** [TokForge on GitHub](https://github.com/darkmaniac7/Elysium)
+
+## Export Details
+
+Converted using MNN's `llmexport` pipeline:
+```bash
+python llmexport.py --path cognitivecomputations/Dolphin-Mistral-24B-Venice-Edition --export mnn --quant_bit 4 --quant_block 128
+```

config.json

@@ -0,0 +1,10 @@
+{
+    "llm_model": "llm.mnn",
+    "llm_weight": "llm.mnn.weight",
+    "backend_type": "cpu",
+    "thread_num": 4,
+    "precision": "low",
+    "memory": "low",
+    "sampler_type": "penalty",
+    "penalty": 1.1
+}

embeddings_bf16.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a135f2a37d2fe0d27956dfacd6e1f7cf8872ba4def4895bd74d1fa35a3830cd6
+size 1342177280

export_args.json

@@ -0,0 +1,42 @@
+{
+    "path": "/root/models/hf_convert_queue/Dolphin-Mistral-24B-Venice-Edition",
+    "type": null,
+    "tokenizer_path": "/root/models/hf_convert_queue/Dolphin-Mistral-24B-Venice-Edition",
+    "eagle_path": null,
+    "lora_path": null,
+    "gptq_path": null,
+    "dst_path": "/root/models/hf_uploads/Dolphin-Mistral-24B-Venice-Edition-MNN",
+    "verbose": false,
+    "test": null,
+    "export": "mnn",
+    "onnx_slim": false,
+    "quant_bit": 4,
+    "quant_block": 128,
+    "visual_quant_bit": null,
+    "visual_quant_block": null,
+    "lm_quant_bit": 4,
+    "lm_quant_block": 128,
+    "mnnconvert": "../../../build/MNNConvert",
+    "ppl": false,
+    "awq": false,
+    "hqq": false,
+    "omni": false,
+    "transformer_fuse": false,
+    "group_conv_native": false,
+    "smooth": false,
+    "sym": false,
+    "visual_sym": false,
+    "seperate_embed": true,
+    "lora_split": false,
+    "calib_data": null,
+    "act_bit": 16,
+    "embed_bit": 16,
+    "act_sym": false,
+    "quant_config": null,
+    "generate_for_npu": false,
+    "skip_weight": false,
+    "omni_epochs": 20,
+    "omni_lr": 0.005,
+    "omni_wd": 0.0001,
+    "tie_word_embeddings": false
+}

llm.mnn

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:986370f65a2f1df7e98f2380e985f8706c3a4b4656dd5064dff382d283b5e756
+size 695576

llm.mnn.weight

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:85c19ecb7096835a7c66a4af3e9b7e524982f1679790d4aedf19132fd39b096f
+size 12885080106

llm_config.json

@@ -0,0 +1,11 @@
+{
+    "model_type": "mistral",
+    "hidden_size": 5120,
+    "attention_mask": "float",
+    "attention_type": "full",
+    "is_mrope": false,
+    "jinja": {
+        "chat_template": "{% for message in messages %}{%if message[\"role\"]==\"system\"%}<|im_start|>system\n{{message[\"content\"]}}<|im_end|>\n{% elif message[\"role\"]==\"user\"%}<|im_start|>user\n{{message[\"content\"]}}<|im_end|>\n{% elif message[\"role\"]==\"assistant\"%}<|im_start|>assistant\n{{message[\"content\"]}}<|im_end|>\n{% endif %}{% endfor %}{% if add_generation_prompt %}<|im_start|>assistant\n{% endif %}",
+        "eos": "<|im_end|>"
+    }
+}