ANLP Assignment 1: Custom Transformers & Byte Latent Transformers (BLT)
Author: Sparsh Bhartia (Roll: )
Institution: International Institute of Information Technology, Hyderabad (IIIT-H)
WandB Project: sparsh-bhartia/anlp-a1-transformers
This repository hosts the trained PyTorch checkpoints (), tokenizers, and evaluation results for 5 architectural configurations (C1 through C5) evaluated on decrypting 8-bit periodic XOR ciphertexts back to clean English plaintext.
Checkpoint Files Included
- ****: Base Seq2Seq Transformer (Sinusoidal PE, Multi-Head Attention, LayerNorm, Learned BPE).
- ****: Rotary Position Embedding (RoPE) Seq2Seq Transformer.
- ****: Grouped-Query Attention (GQA) Seq2Seq Transformer (=4, H_{KV}=2$).
- ****: Root Mean Square Normalization (RMSNorm Pre-LN) Seq2Seq Transformer.
- ****: Byte Latent Transformer (BLT) with Entropy-Based Dynamic Patching.
Supporting Artifacts
- : Learned BPE tokenizer for ciphertext.
- : Learned BPE tokenizer for English plaintext.
- : 3-gram byte entropy model for BLT dynamic patching.
- : Full benchmark evaluation results.
Benchmark Results
| Model | Bit-Acc (%) | Seq-Acc (%) | Lev Dist | BLEU-4 | ROUGE-L |
|---|---|---|---|---|---|
| C1 (Base) | 98.42 | 74.80 | 4.12 | 86.45 | 91.20 |
| C2 (RoPE) | 99.15 | 81.20 | 2.45 | 89.78 | 93.65 |
| C3 (GQA) | 98.10 | 72.40 | 4.88 | 85.12 | 90.45 |
| C4 (RMSNorm) | 98.50 | 75.60 | 3.95 | 86.90 | 91.55 |
| C5 (BLT Dynamic) | 97.85 | 68.40 | 5.62 | 81.30 | 87.40 |
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support