adding readme

README.md

@@ -13,3 +13,66 @@ tags:
 - transformers
 - mlx
 ---
+# Update
+
+Added a Jinja chat template so the model can format conversations correctly and work smoothly with `mlx-lm` chat-style inference.
+
+# MLX 4-Bit Quantized: Gemma-4-12B-Coder
+
+This repository contains an 4-bit MLX-converted version of
+[yuxinlu1/gemma-4-12B-coder-fable5-composer2.5-v1](https://huggingface.co/yuxinlu1/gemma-4-12B-coder-fable5-composer2.5-v1).
+
+The model has been quantized to 4-bit to dramatically reduce memory requirements while retaining near-lossless reasoning and coding capabilities. It is optimized for local inference on Apple Silicon Macs using the `mlx-lm` library.
+
+## How to Use with MLX
+
+Install the required dependency:
+
+```bash
+pip install --upgrade mlx-lm
+```
+
+Run inference from Python:
+
+```python
+from mlx_lm import load, generate
+
+# Load the 8-bit quantized MLX model.
+model, tokenizer = load("mlx-community/gemma-4-12b-coder-fable5-composer2.5-4bit")
+
+prompt = "Write a Python script to sort a dictionary by its values."
+messages = [{"role": "user", "content": prompt}]
+
+formatted_prompt = tokenizer.apply_chat_template(
+    messages,
+    tokenize=False,
+    add_generation_prompt=True,
+)
+
+response = generate(
+    model,
+    tokenizer,
+    prompt=formatted_prompt,
+    verbose=True,
+    max_tokens=1024,
+)
+```
+
+```python
+response = generate(
+    model,
+    tokenizer,
+    prompt=formatted_prompt,
+    verbose=True,
+    max_tokens=1024,
+    temp=0.0,
+)
+```
+
+## Base and License
+
+- **Base model:** `google/gemma-4-12B-it`
+- **Original fine-tune:** [`yuxinlu1/gemma-4-12B-coder-fable5-composer2.5-v1`](https://huggingface.co/yuxinlu1/gemma-4-12B-coder-fable5-composer2.5-v1)
+- **License:** Apache 2.0
+
+Free to use, modify, and redistribute under the Apache 2.0 license.