Wall Attention - Length Generalization with Diagonal Gates

• Attention is permutation equivariant and can't determine temporal effects on its own
• Positional embeddings like rope and alibi inject this temporal information to enable recency bias
• can break approaches to encoding position into data dependent and independent
• rope would fall into the data-independent category because the positional encoding doesn't change based on the data - you always do the rotations in the same way
• there have been data-dependent methods like FoX (forgetting attention) which decide how to modulate positional prior per token
• other dimension is additive bias vs multiplicative
• wall attention capable of selectively remembering and forgetting per channel

This will be a good article to come back to.