Scaling Context to 128k: RoPE, YaRN & Needle-in-a-Haystack Benchmarks

Extending an LLM's context window from 8,192 tokens to 128,000 tokens sounds simple on paper: just adjust the sequence length flag. But in practice, naive context extension results in severe attention entropy collapse where the model starts hallucinating or completely forgets facts buried in the middle of documents.

1. The Problem: Attention Entropy Collapse

Rotary Position Embedding (RoPE) encodes word position as rotation angles in complex space. When you feed a model 50,000 tokens when it was only trained on 8,000, the rotation angles reach unseen frequencies. The dot-product attention scores explode, causing the softmax distribution to become uniformly flat.

2. YaRN vs Linear Scaling Math

YaRN (Yet another RoPE extensioN) solves this by dividing frequencies into three bands: high frequencies (local syntax) are left untouched, low frequencies (long-range semantic position) are interpolated, and middle frequencies are smoothed using a ramp function.

3. 128k Needle-in-a-Haystack Test Results

We ran an automated Needle-In-A-Haystack (NIAH) test by inserting a secret UUID passkey at 10% depth increments across 128,000 tokens of random RFC security documentation:

• Default Linear RoPE (scale=16.0): 100% recall up to 16k tokens, drops to 34% recall at 64k, and complete failure (0%) at 128k.
• YaRN RoPE (scale=16.0, beta_fast=32, beta_slow=1): 99.4% recall across all depth positions up to the full 128,000 token limit.