RoPE Decay Curve Is Noise: Taking Absolute Values Won't Fix It

Rotary Position Embedding (RoPE) is widely used in Transformers: it injects the absolute position of each token into queries and keys, yet the attention score between any two tokens depends only on the distance between them. Shift an entire sentence ten tokens back, and the internal scores stay completely unchanged. This counterintuitive property is RoPE's core selling point—and the starting point for many misreadings of it.

This article is based on the complete derivation, experimental data, and verification process of an interactive technical blog, and breaks down three common misunderstandings: why the decay curve is noise, why taking absolute values doesn't save it, and why NTK scaling can't touch the fastest frequency pair. The original post provides a demo page that runs directly in the browser, with every conclusion backed by reproducible code.

A One-Line Identity: Absolute Position Appears Only as a Difference

Write the 2D component of the query at position m as a complex number q, and the key at position n as k. RoPE multiplies them by e^(imθ) and e^(inθ) respectively, so:

Re[ (q·e^(imθ)) · conj(k·e^(inθ)) ] = Re[ q·conj(k)·e^(i(m−n)θ) ]

Absolute position survives only in the exponent, in the form (m−n). The original post verified this across 42 offset-and-shift combinations, with shifts up to 4000, and a maximum deviation of 1e−15—floating-point error territory. As a contrast, additive sinusoidal positional encoding gives different scores for the same offset at different absolute positions—which shows that RoPE's "relativity" is not something every positional encoding has.

The original site also gives a minimal JavaScript implementation that rotates each 2D component by pos·θ in pairs—logically, the complex multiplication above expanded into real form.

Additive sinusoidal positional encoding is the positional encoding scheme used in the original Transformer paper, proposed by Vaswani et al. in 2017. It adds the sine and cosine values of the position directly to the input word embeddings: PE(pos, 2i) = sin(pos / 10000^(2i/d)), PE(pos, 2i+1) = cos(pos / 10000^(2i/d)). This encoding superimposes positional information as a signal onto token representations, and every subsequent attention layer re-encounters these encoded absolute positions. The key difference from RoPE is: RoPE applies positional rotation to the phases of queries/keys, so the absolute quantity naturally cancels in the dot product, leaving only the relative difference. Additive encoding, after the dot product is expanded, retains cross terms tied to absolute position, so the same token pair gets different scores at different absolute positions—it lacks RoPE's exact relativity.

Dephasing is borrowed from quantum mechanics and describes how a set of coherent oscillations gradually loses phase alignment because of differing frequencies. In RoPE, the dot product of a fixed vector with itself at different distances can be seen as a weighted sum of components cos((m−n)θᵢ) at different frequencies. As distance grows, the phase differences of the components spread out, positive and negative terms cancel, and the expected magnitude of the sum drops—this is the true source of "decay." It is not decay of vector length or attention magnitude, but degradation of phase alignment, which is why amplitude-only tools like absolute value cannot capture or restore it.

Why the Decay Curve Is Noise

The most widely circulated claim is that "attention under RoPE decays with distance." But if you simply plot each token pair's attention score against distance, you get a flat line made of random scatter, not a declining curve. The reason is plain: the expected dot product of two independent random vectors is zero at any distance. The curve you draw is actually sampling noise.

The original post lays this out with a three-column comparison table: fix the same set of vectors and the same offsets, then look at "compared with itself," "mean raw score over 600 random pairs," and "mean absolute value over the same pairs."

offset	q vs. itself	Mean raw score (600 pairs)	Mean |score|
0	10.60	−0.07	3.17
16	8.05	−0.37	3.12
256	4.95	−0.28	3.04
1024	0.69	−0.38	3.02

The three columns point to the same fact: the mean raw score never leaves ±0.38, while the typical magnitude is 3.2; the mean absolute value moves only 4% across the whole distance range. The only column that truly collapses is "compared with itself."

This is what people really mean by "decay": a fixed vector's phase versus itself gradually drifts out of alignment at different distances, and the cosines cancel each other out. It is dephasing, not amplitude attenuation. Measuring with random vectors is equivalent to throwing away the signal itself.

Why Taking Absolute Values Can't Save It

After finding that the raw mean is zero, the common fix is to look at magnitude. The third column already gives the answer: the mean absolute value slips only from 3.17 to 3.02 between offset=0 and 1024. If there were a real distance-decay mechanism, there should be a monotonic and significant drop here, but the actual change is at noise level.

The reason is that absolute value loses phase-direction information: two vectors that are perfectly out of phase cannot be distinguished in magnitude from one vector in phase, and decay is precisely a phase phenomenon.

"Distance decay" describes the self-similarity of a fixed vector, not a statistical law of attention weights for random pairs. Confusing these two is the source of most faulty measurements.

The Random Number Bug That Produced Wrong Measurements

The original post confesses an even subtler pitfall. Initially, each "random" vector was produced by an independent Park-Miller generator, with seeds taken from neighboring integers. The problem: neighboring LCG seeds produce highly correlated sequences.

The consequence was that samples that should have been independent and zero-mean drifted to ±2.5—an order of magnitude above the standard error—and looked like some kind of real structure. After switching to a single generator shared by the whole experiment, the mean immediately returned to ±0.38.

The lesson is direct: if your "independent samples" come from rng(seed + i), they are not independent. This is a high-frequency trap in numerical experiments, especially easy to miss with small samples or when serial correlation has not been tested.

Park-Miller generator (also called the minimal standard LCG) is a linear congruential pseudorandom number generator: xₙ₊₁ = (16807 · xₙ) mod (2³¹ − 1). It is simple to implement and fast, but has a fatal weakness: when seed values are close to one another, the generated sequences remain highly correlated in the first several terms. This is because an LCG is essentially a fixed-step multiplication jump in modulus space; neighboring seeds move in clusters in their initial trajectories until enough steps have passed for mixing to occur.

LCG (linear congruential generator) behavior can be understood this way: every output is produced by multiplying the state by a constant and taking a modulus, so nearby points in state space maintain a nearby "geometric" relationship in the short term. Worse, the low-order bits of an LCG have extremely short periods, so taking xₙ mod a small integer directly as a random number exposes obvious periodic structure.

In numerical experiments, seeding each sample separately with rng(seed + i) is equivalent to starting adjacent samples from almost identical initial conditions, so the output sequences are highly correlated. This artificially lowers or raises the variance of the average of samples that should be independent, and the mean drifts outside the standard error, looking like the discovery of some "real structure." The correct approach is to sample sequentially from a single generator for the whole experiment, or to use methods such as the Marsaglia polar method that mix multiple seed outputs again, avoiding correlation between seeds contaminating sample independence.

NTK Scaling Can't Touch the Fastest Pair

Another common confusion comes from length extrapolation: training to 512, inference asking for 4096—which frequency pair actually breaks? Intuition blames the "fast-spinning" high-frequency pair, but those have already looped around many times within the training window; another twenty loops is nothing new.

What truly leaves familiar territory is the slowest pair. Its period is longer than the whole context, and during training it sweeps only 2.6% of a cycle; after positions are scaled up eightfold, it enters an angle range never seen during training for the first time.

Strategy	Phase angle of slowest pair	Notes
Training, position 512	0.162 rad	What the model has seen
Untreated, position 4096	1.295 rad	8× outside the training distribution
Position interpolation	0.162 rad	Exactly back to the training point
NTK-aware base	0.162 rad	Exactly back to the training point

Both repair methods send it back to the same position at the slowest end. The difference appears at the fastest end: θ₀ = base⁰ = 1, unchanged no matter how base is adjusted, so NTK scaling cannot slow down the fastest frequency pair at all; position interpolation slows all frequencies uniformly, at the cost of compressing the resolution between adjacent tokens.

This also explains why the two methods perform differently on different tasks: their ways of handling high and low frequencies are not equivalent.

Neural Tangent Kernel (NTK) theory was proposed by Jacot et al. in 2018, showing that the evolution of an infinitely wide neural network under gradient descent training can be equivalent to a kernel method, with the kernel determined by the Jacobian matrix at network initialization. In the context of positional encoding extrapolation, the term "NTK-aware" is borrowed and adapted: it does not directly use the original NTK, but borrows the intuition that "different frequency components should be adjusted at different speeds."

Each frequency dimension θᵢ = base^(−2i/d) in RoPE corresponds to a rotation speed. During training, low-frequency dimensions rotate slowly and do not cover a full period within the window; high-frequency dimensions rotate quickly and loop repeatedly. When inference length expands, low-frequency dimensions enter a phase range never seen in training for the first time, which is the main cause of extrapolation failure. The "spectral bias" in original NTK theory tells us that neural networks tend to learn low-frequency functions first—this provides theoretical background for "low-frequency dimensions need more protection during length extrapolation." NTK-aware scaling therefore chooses to increase the base value (e.g., from 10000 to 1000000), making the periods of all frequencies longer overall, so that while high-frequency rotation is kept unchanged, low-frequency dimensions are pulled back to trained phases. But as the original post points out, the fastest frequency θ₀ = base⁰ = 1 does not change with base at all, so NTK scaling has no effect on it; its control is concentrated at the low-frequency end.

Position Interpolation takes another approach: it does not change base, but compresses the inference position indices overall, for example mapping 4096 back into a range below 512. In this way the phase of every frequency is scaled by the same ratio, all frequencies are "slowed," and the phase difference between adjacent tokens is uniformly compressed. The cost is that the model's resolution for distinguishing token positions over short distances decreases, potentially harming short-range dependency modeling. The asymmetry between the two in handling high and low frequencies is exactly the source of their different performance across tasks.

Verification Methods and a Few Key Conclusions

The original verifier did three things: compute with complex multiplication rather than the real 2×2 form; extract the rotation as a matrix and check RᵀR = I and det = +1; and remeasure decay with a Marsaglia polar sampler to avoid reproducing the page's own generator bug during verification. This cross-check even caught a sign error in the author's closed-form solution—the sin term should have been − but was written as +.

The final result is 226 verification assertions, 28 in-page checks, 0 failures.

Bringing the whole thread together:

• RoPE's absolute position appears only as a difference; across 42 test groups the maximum deviation is 1e−15, and this is its mathematical foundation.
• The attention score of a random vector pair has expectation zero at any distance, so plotting it directly yields noise; "decay" describes the phase dephasing of a fixed vector with itself.
• Taking absolute values cannot recover the decay signal, because phase-direction information has already been lost.
• Sample correlation caused by neighboring LCG seeds can push the mean from ±0.38 to ±2.5, and is the direct cause of the initial faulty measurement.
• The first thing to fail in length extrapolation is the slowest frequency pair (which sweeps only 2.6% of a cycle during training). NTK scaling works on it but is completely powerless for the fastest frequency pair.

Next time you see a smooth "RoPE decay curve," first confirm whether its sampling target is a fixed vector or random pairs, and whether the random number generator is shared. These two points determine whether you are seeing a physical law or sampling noise.

Marsaglia polar sampler is a classic method for generating standard normally distributed random numbers. It uniformly samples a point (u, v) inside the unit circle, discards samples outside the circle or at the zero point, then lets s = u² + v², and outputs z₁ = u · √(−2 ln s / s) and z₂ = v · √(−2 ln s / s). These two outputs are independent and follow the standard normal distribution.

Compared with the Box-Muller method (which starts from polar angle and radius), the Marsaglia method avoids trigonometric calculations, is faster, and has better numerical stability, because it directly uses the geometric properties of uniform sampling inside a circle to construct the normal distribution. In the original verifier, the Marsaglia polar sampler was used to remeasure decay in order to bypass possible LCG seed correlation in the page's own generator, and to cross-validate the authenticity of the decay data with a completely different and well-tested random number generation mechanism. If two independent implementations give the same statistical conclusion, then the originally observed phenomenon is not an artifact of a particular generator. This practice reflects a principle in numerical experiments: key conclusions should be reproduced along independent implementation paths, not merely be self-consistent within the same potentially flawed toolchain.

One-line Summary

RoPE's attention decay curve is actually sampling noise; the root cause is phase dephasing of a fixed vector, not amplitude decay of random pairs.

Paragraph Summary

This article breaks down three common misunderstandings about Rotary Position Embedding (RoPE). First, the expected dot product of random vector pairs is zero at any distance, so a directly plotted decay curve is only sampling noise; the real "decay" comes from phase dephasing of a fixed vector with itself at different distances, while the mean absolute value barely changes, so taking absolute values cannot fix it. Second, neighboring LCG seeds produce highly correlated random sequences, which once caused experimental means to drift from ±0.38 to ±2.5, the direct source of faulty measurements. Finally, in length extrapolation, the slowest frequency pair fails first (sweeping only 2.6% of a cycle during training); NTK scaling can pull it back to the training phase, but is completely ineffective for the fastest frequency pair; position interpolation slows all frequencies uniformly, at the cost of compressed short-range resolution. The verifier uses complex multiplication, matrix orthogonality checks, and an independent polar sampler for cross-validation, and all 226 assertions pass.