Config 4: GQA + RoPE + RMSNorm Transformer
This model is part of the ANLP Assignment 1: Custom Transformers & Byte Latent Transformers at IIIT Hyderabad.
Architecture
- Description: Encoder-Decoder Seq2Seq Transformer, Grouped Query Attention (GQA, 8 query heads, 2 KV groups), RoPE, RMSNorm.
- Parameters: 11.18M
Test Evaluation Results
- Bit-Level Accuracy: 93.58%
- Sequence Accuracy: 68.02%
- BLEU Score: 98.77%
- ROUGE-1 / ROUGE-2 / ROUGE-L: 99.35% / 98.15% / 99.35%
- Average Levenshtein Distance: 0.41
Repository & Links
- GitHub: https://github.com/SinisterLlamma/ANLP-Assignment1
- Author: Eshaan Sharma (SinisterLlama)
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support