robertobissanti/EngGPT2-16B-A3B-MLX-8bit

EngGPT2-16B-A3B MLX 8-bit

This repository contains an 8-bit MLX conversion of engineering-group/EngGPT2-16B-A3B. This version is larger than the 4-bit conversion and may preserve more numerical precision, but in local tests the 4-bit version provided the best practical speed/memory tradeoff.

Important compatibility note

This model currently requires custom enggpt_moe support in mlx-lm. Standard upstream mlx-lm may not load this model unless support for:

model_type = enggpt_moe

has been merged.

A patched mlx-lm fork is available here:

https://github.com/robertobissanti/mlx-lm-enggpt

A copy of the custom model implementation is also included in this repository:

custom_mlx/enggpt_moe.py

Until upstream support is available, users must either:

1. install the patched fork, or
2. copy custom_mlx/enggpt_moe.py into their local mlx_lm/models/ directory.

Original model

Base model:

engineering-group/EngGPT2-16B-A3B

The original model is a custom MoE decoder-only language model.

Approximate architecture:

• 24 decoder layers
• hidden size 2880
• 32 attention heads
• 4 key/value heads
• explicit head_dim = 128
• Q/K RMSNorm inside attention
• RoPE with rope_theta = 1000000.0
• 64 experts per MoE layer
• top-8 expert routing
• SwiGLU experts
• untied lm_head

Tested setup

Tested locally on:

• Mac Studio M1 Ultra
• 64 GB unified memory
• macOS Apple Silicon
• patched mlx-lm fork: robertobissanti/mlx-lm-enggpt Approximate local benchmark:

Version Disk size Generation speed Peak memory MLX 8-bit ~20 GB ~75–80 tok/s ~21.3 GB

Installation

Install the patched mlx-lm fork:

python3 -m venv .venv-mlx-enggpt
source .venv-mlx-enggpt/bin/activate
pip install -U pip
pip install git+https://github.com/robertobissanti/mlx-lm-enggpt.git

Download

hf download robertobissanti/EngGPT2-16B-A3B-MLX-8bit \
  --local-dir EngGPT2-16B-A3B-MLX-8bit

Usage

  --model ./EngGPT2-16B-A3B-MLX-8bit \
  --prompt "Spiegami in italiano, in due frasi, che cos'è un modello Mixture of Experts." \
  --trust-remote-code \
  --chat-template-config '{"enable_thinking": false}' \
  --temp 0.1 \
  --max-tokens 160```
  
## Open WebUI example

You can run this model through mlx_lm.server and connect Open WebUI to the local OpenAI-compatible endpoint.

Start the MLX server:

python -m mlx_lm server
--model ./EngGPT2-16B-A3B-MLX-8bit
--host 127.0.0.1
--port 8080
--trust-remote-code
--chat-template-args '{"enable_thinking": false}'
--temp 0.1
--top-p 1.0
--max-tokens 512

Start Open WebUI:

docker run -d
--name open-webui
-p 3000:8080
-e WEBUI_AUTH=false
-e OPENAI_API_BASE_URL=http://host.docker.internal:8080/v1
-e OPENAI_API_KEY=dummy
-v open-webui:/app/backend/data
--restart unless-stopped
ghcr.io/open-webui/open-webui:main

Then open:

http://localhost:3000/

When configuring the model, use the local model path as the model id, for example:

./EngGPT2-16B-A3B-MLX-8bit

or the absolute local path used when starting the server.
  

## Chat template note

The tokenizer/chat template may emit <think> blocks unless thinking is disabled.

Use:

```{"enable_thinking": false}```

With mlx_lm.generate:

--chat-template-config '{"enable_thinking": false}'

With mlx_lm.server:

--chat-template-args '{"enable_thinking": false}'

## Known limitations

* Requires custom enggpt_moe support in mlx-lm.
* Does not currently work in standard LM Studio unless its backend includes enggpt_moe.
* Does not currently work in Ollama because no GGUF conversion exists.
* llama.cpp conversion currently fails with:

Model EngGPTMoeForCausalLM is not supported

* This is an experimental community conversion.

## License

This repository is a derived MLX conversion of:

[engineering-group/EngGPT2-16B-A3B](https://huggingface.co/engineering-group/EngGPT2-16B-A3B)

Please refer to the original model repository for license terms and usage restrictions.

The original model is distributed under the EngGPT Non-Commercial License. Commercial use may be restricted or prohibited by the original license.