---
language:
- en
tags:
- mlp-mixer
- causal-lm
- byte-level
- attention-free
- hypermixing
datasets:
- mookiezi/Discord-Dialogues
license: apache-2.0
---
---
## 📋 Overview
**MicroMixer-2-500K-discord-dialogues** is a ~779K parameter MLP-Mixer language model trained on Discord conversation data. With 4 mixer layers and larger hidden dimensions than the 300K variant, it offers improved capacity while remaining compact. Features DropPath regularization and label smoothing for stable training.
---
## 🏗️ Architecture
```mermaid
graph TD
A[Byte Input] --> B[Token Embedding]
B --> C[RoPE Position Encoding]
C --> D[MicroMixerLayer ×4]
D --> E[LayerNorm]
E --> F[LM Head]
F --> G[Byte Output]
style A fill:#007BFF,color:#fff
style G fill:#00D620,color:#fff
style D fill:#AE00FF,color:#fff
```
### Model Configuration
| Parameter |
Value |
| Total Parameters | 778,752 |
| Hidden Dimension | 176 |
| Hyper Hidden Dimension | 88 |
| Channel MLP Dimension | 384 |
| Number of Layers | 4 |
| Max Sequence Length | 128 |
| Vocabulary Size | 256 (Byte-level) |
| DropPath Rate | 0.1 |
| Label Smoothing | 0.05 |
### Core Components
```
┌─────────────────────────────────────────────┐
│ MicroMixerLayer │
│ ┌─────────────────────────────────────┐ │
│ │ LayerNorm → HyperMixing → Residual │ │ ← Token Mixing
│ ├─────────────────────────────────────┤ │
│ │ LayerNorm → MlpBlock → Residual │ │ ← Channel Mixing
│ └─────────────────────────────────────┘ │
└─────────────────────────────────────────────┘
```
#### 1️⃣ RoPE (Rotary Position Embedding)
- Encodes positions via **rotation transformations**
- Enables length extrapolation beyond training sequences
#### 2️⃣ HyperMixing (Token Mixing)
- Compresses past context via **cumulative average pooling**
- Hypernetwork generates adaptive weights
- O(S) complexity token mixing without attention
#### 3️⃣ MlpBlock (Channel Mixing)
- Non-linear transformation of feature dimensions
- Structure: `Linear → GELU → Linear`
#### 4️⃣ V4 Innovations
- **DropPath**: Stochastic depth regularization (random residual skipping)
- **Label Smoothing**: Prevents overconfident predictions
- **Padding-Aware Loss**: Ignores padding tokens in cross-entropy
---
## 🎯 Generation Examples
```
[Prompt] User: Hello
[Output] Assistant:
User: Good why are you good that
Assistant: What to mors of reallDamablin
[Prompt] User: How are you?
[Output] Assistant: I will good to can just a hond slee like to be you bany too the the moustouch me a tan moll troove
[Prompt] User: What is your name?
[Output] Assistant:
User: wait you saying suppering bro
Assistant: I was really to move your an a planers tot whin is
```
---
## 📊 Training Results
| Metric | Value |
|--------|-------|
| Train Loss | 2.0141 |
| Train PPL | 7.49 |
| Val Loss | 1.9101 |
| Val PPL | 6.75 |
| Epoch | 3 |
| Global Steps | 42,186 |
**Training Time**: ~194 seconds per epoch
---
## 📊 Training Data
**Dataset**: [Discord-Dialogues](https://huggingface.co/datasets/mookiezi/Discord-Dialogues)
- 500,000 Discord conversation samples (subset)
- Converted from ChatML to User/Assistant format
- Multi-turn conversational data
- Sequence length: 128 tokens
---
## 🔧 Usage
```python
import torch
from huggingface_hub import hf_hub_download
from src.model import MicroMixer, MicroMixerConfig
from src.tokenizer import ByteTokenizer
# Clone the repository first:
# git clone https://github.com/llaa33219/MicroMixer-2.git
# cd MicroMixer-2
config = MicroMixerConfig(
max_seq_len=128,
hidden_dim=176,
hyper_hidden_dim=88,
channel_mlp_dim=384,
num_layers=4,
)
model = MicroMixer(config)
weights_path = hf_hub_download("llaa33219/MicroMixer-2-500K-discord-dialogues", "model.pt")
model.load_state_dict(torch.load(weights_path, map_location="cpu"))
model.eval()
tokenizer = ByteTokenizer()
input_ids = torch.tensor([tokenizer.encode("User: Hello
Assistant:")])
with torch.no_grad():
output = model.generate(input_ids, max_new_tokens=64, temperature=0.7, top_k=40)
print(tokenizer.decode(output[0].tolist()))
```
---
---
## ⚠️ Limitations
| Limitation | Description |
|------------|-------------|
| **Small Model** | Only ~779K parameters |
| **Limited Context** | Max 128 tokens sequence length |
| **Grammar Issues** | Generated text has grammatical errors |
| **Coherence** | Output lacks coherent conversation flow |
| **Limited Training** | Only 3 epochs on 500K samples |
---
[](https://github.com/llaa33219/MicroMixer-2)
Part of the MicroMixer-2 research project
# MicroMixer-2-500K-discord-dialogues
