Rotary Position Embeddings (RoPE)

RoPE is a popular Relative Positional Encoding implementation which is used in Large Language Model like LLaMA.

In order to generalize our results in 2D to any $x_i\in {\mathbb{R}}^d$ where $d$ is even, we divide the $d$-dimension space into $d/2$ sub-spaces and combine them in the merit of the linearity of the inner product, turning $f_{\{q,k\}}$ into:

$$f_{\{q,k\}}(x_m,m)=R_{\Theta ,m}^d{W}_{\{q,k\}}{x}_m$$

where

$$R_{\Theta ,m}^d=\left(\begin{array}{ccccccc}\cos m{\theta }_1& -\sin m{\theta }_1& 0& 0& \cdots & 0& 0\\ \sin m{\theta }_1& \cos m{\theta }_1& 0& 0& \cdots & 0& 0\\ 0& 0& \cos m{\theta }_2& -\sin m{\theta }_2& \cdots & 0& 0\\ 0& 0& \sin m{\theta }_2& \cos m{\theta }_2& \cdots & 0& 0\\ ⋮& ⋮& ⋮& ⋮& \ddots & ⋮& ⋮\\ 0& 0& 0& 0& \cdots & \cos m{\theta }_{d/2}& -\sin m{\theta }_{d/2}\\ 0& 0& 0& 0& \cdots & \sin m{\theta }_{d/2}& \cos m{\theta }_{d/2}\end{array}\right)$$

is the rotary matrix with pre-defined parameters $\Theta =\{{\theta }_i=10000^{-2(i-1)/d},i\in [1,2,\dots ,d/2]\}$. A graphic illustration of RoPE is shown in Figure (1). Applying our RoPE to self-attention in Equation (2), we obtain:

$$q_m^T{k}_n={\left(R_{\Theta ,m}^d{W}_q{x}_m\right)}^T\left(R_{\Theta ,n}^d{W}_k{x}_n\right)=x^T{W}_q{R}_{\Theta ,m}^d{R}_{\Theta ,n-m}^d{W}_k{x}_n$$

where $R_{\Theta ,n-m}^d={\left(R_{\Theta ,m}^d\right)}^T{R}_{\Theta ,n}^d$. Note that $R_{\Theta }^d$ is an orthogonal matrix, which ensures stability during the process of encoding position information.

---

Reference