Config 3: RoPE + RMSNorm Transformer
This model is part of the ANLP Assignment 1: Custom Transformers & Byte Latent Transformers at IIIT Hyderabad.
Architecture
- Description: Encoder-Decoder Seq2Seq Transformer, Rotary Position Embeddings (RoPE), RMSNorm, MHA.
- Parameters: 11.67M
Test Evaluation Results
- Bit-Level Accuracy: 93.58%
- Sequence Accuracy: 68.11%
- BLEU Score: 98.88%
- ROUGE-1 / ROUGE-2 / ROUGE-L: 99.41% / 98.20% / 99.41%
- Average Levenshtein Distance: 0.42
Repository & Links
- GitHub: https://github.com/SinisterLlamma/ANLP-Assignment1
- Author: Eshaan Sharma (SinisterLlama)
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support