The paper shows Transformers outperform LSTMs on translation, but there's no discussion of inference efficiency. For sequences longer than 8K tokens, the O(n²) attention complexity becomes prohibitive. Flash Attention solves this, but it wasn't known in 2017. Has anyone benchmarked vanilla attention vs FlashAttention-2 at context lengths of 32K-100K? What's the practical crossover point for different hardware?

The positional encoding scheme in the paper uses fixed sinusoidal functions. This fundamentally limits generalization to sequence lengths seen during training. ALiBi, RoPE, and YaRN all address this differently. Which works best in practice for lengths significantly beyond training distribution? Any ablation data?