---
license: apache-2.0
library_name: pytorch
pipeline_tag: text-generation
tags:
  - text-generation
  - pytorch
  - gpt
  - zenyx
  - transformer
  - from-scratch
  - causal-lm
  - custom-architecture
language:
  - en
datasets:
  - HuggingFaceFW/fineweb-edu
---
---
base_model: null
model_creator: Arko007
model_version: v1.0
model_date: 2025-10
model_card_authors:
  - Arko007
---

# 💠 Zenyx-42M: Where Calm Meets Power

Zenyx-42M is a 42M parameter GPT-2 style decoder-only transformer **trained from scratch** on high-quality educational web text.

The name "Zenyx" fuses **Zen** (calm, focused intelligence) and **Onyx** (strength, power)—embodying an efficient, capable language model.

---

## Model Details

| Component          | Value                                        |
|--------------------|----------------------------------------------|
| Architecture       | GPT-2 style decoder-only transformer         |
| Parameters         | ~42M (41.87M)                                |
| Layers             | 8                                            |
| Hidden Size        | 512                                          |
| Attention Heads    | 8                                            |
| Context Length     | 512 tokens                                   |
| Vocabulary Size    | 32,000 (BPE)                                 |
| Positional Encoding| Learned embeddings                           |

### Training Configuration

| Setting            | Value                                        |
|--------------------|----------------------------------------------|
| Training Data      | FineWeb-Edu (streamed, deduped, filtered)    |
| Training Mode      | Scratch (no pretrain)                        |
| Optimizer          | AdamW (lr=3e-4, wd=0.1)                      |
| LR Schedule        | Warmup (2k) + Cosine decay                   |
| Batch Size         | 16 per device, 8 grad. accumulation          |
| Effective Batch    | 128                                          |
| Precision          | BFloat16 mixed precision                     |
| Hardware           | NVIDIA L4 (24GB VRAM)                        |
| Training Time      | ~6 hours                                     |
| Iterations         | 100,000                                      |
| Tokens Processed   | ~6.55B                                       |

---

## Performance

| Model           | Params | Val Loss | Zenyx Advantage   |
|-----------------|--------|----------|-------------------|
| **Zenyx-42M**   | 42M    | 3.08     | Baseline          |
| GPT-1           | 117M   | ~3.3     | +22% better       |
| DistilGPT-2     | 82M    | ~3.1     | Nearly tied       |
| GPT-2 Small     | 124M   | ~2.8     | 3x more efficient |

- **Training Loss**: 0.398
- **Validation Loss**: 3.08
- **Tokens Processed**: ~6.55B

---

## Capabilities

- ✅ Text completion
- ✅ Topic consistency
- ✅ Grammar & syntax
- ✅ Technical vocabulary
- ⚠️ Base model (fine-tuning required for instructions)
- ⚠️ 512 token context

---

## Usage

**Install dependencies**  
```bash
pip install torch transformers tokenizers
```

**Quick Start Example**
```python
import torch
from model import NanoGPT
from config import NanoGPTConfig
from tokenizers import Tokenizer

# Load model
checkpoint = torch.load('best_model.pt', map_location='cpu', weights_only=False)
config = checkpoint['config']
model = NanoGPT(config)
model.load_state_dict(checkpoint['model'])
model.eval()

# Load tokenizer
tokenizer = Tokenizer.from_file('tokenizer.json')

def generate(prompt, max_tokens=100, temperature=0.6):
    tokens = tokenizer.encode(prompt).ids
    x = torch.tensor(tokens).unsqueeze(0)
    with torch.no_grad():
        output = model.generate(x, max_new_tokens=max_tokens, temperature=temperature, top_k=40)
    return tokenizer.decode(output.tolist())

prompt = "Artificial intelligence is"
output = generate(prompt)
print(output)
```

---

## Limitations

- ⚠️ Base model only (no instruction fine-tuning)
- ⚠️ Can repeat phrases (use repetition penalty)
- ⚠️ 512 token context window
- ⚠️ Small parameter count limits knowledge
- ⚠️ Possible factual errors
- ⚠️ Not suitable for reliable code generation

**Not recommended for:**
- ❌ Production applications (without fine-tuning)
- ❌ Factual QA
- ❌ Instruction following
- ❌ Mission-critical tasks

---

## Fine-Tuning

Designed as a **base model** for further fine-tuning.

**Next Steps:**
1. Instruction fine-tuning (Alpaca/Dolly datasets)
2. Code specialization (GitHub/StackOverflow)
3. Domain adaptation (medical/legal/etc)
4. RLHF (human feedback alignment)

---

## Ethical Considerations

- Training data may contain bias
- Intended for educational/experimental use
- Not suitable for high-stakes decisions
- Fully open source and documented
- No explicit safety fine-tuning

---

## Citation

```bibtex
@misc{zenyx_42m_2025,
  author = {Arko007},
  title = {Zenyx-42M: Efficient Language Model Trained From Scratch},
  year = {2025},
  month = {October},
  publisher = {Hugging Face},
  url = {https://huggingface.co/Arko007/Zenyx-42M}
}
```

---

## Model Card Authors

- **Developer:** Arko007  
- **Date:** October 2025  
- **Contact:** [HuggingFace Profile](https://huggingface.co/Arko007)

---

## License

Apache License 2.0  
See LICENSE file for details.

---

## Acknowledgments

- HuggingFace FineWeb-Edu team (dataset)
- GPT-2 (OpenAI, architecture inspiration)
- PyTorch (framework)
- NVIDIA L4 GPU (compute)

---

## Version History

- **v1.0** (Oct 2025): Initial release — 42M params, 100k iterations, val loss 3.08

---

## 💠 Branding

**ZENYX**  
Symbol: 💠 diamond/geometric  
Colors: Deep purple (#5B21B6) + Cyan (#06B6D4)  
Font: Modern, clean, geometric  
Taglines:  
- "Where Calm Meets Power" 💪  
- "Efficient Intelligence, Powerful Results" ⚡  
- "Small Model, Big Impact" 💎  
- "Zen Precision, Onyx Strength" 🔷  
Brand Personality:  
- Calm & efficient  
- Powerful  
- Modern  
- Balanced (Zen meets power)  

---

**Built with 💠 using PyTorch and L4 GPU**