title: Why Are Positional Encodings Nonessential for Deep Autoregressive Transformers? A Petroglyph Revisited
abstract: Do autoregressive Transformer language models require explicit positional encodings (PEs)?The answer is 'no' provided they have more than one layer-they can distinguish sequences with permuted tokens without the need for explicit PEs.This follows from the fact that a cascade of (permutation invariant) set processors can collectively exhibit sequence-sensitive behavior in the autoregressive setting.This property has been known since early efforts (contemporary with GPT-2) adopting the Transformer