RULER

A model’s system card might advertise a million-token context window, but RULER exists to ask whether it actually uses that whole window well. Built by researchers at NVIDIA, it extends the basic needle-in-a-haystack idea into 13 synthetic tasks across four categories — plain retrieval with configurable noise and multiple needles, multi-hop variable tracking, aggregation over the entire input, and long-document question answering — run at sequence lengths from 4,000 tokens up past a million. Instead of reporting raw accuracy, RULER derives an “effective context length”: the longest input a model can handle while staying close to its own short-context performance.

The result at launch was a wide gap between advertised and effective context. Of the 17 models tested in the original 2024 paper, Gemini 1.5 Pro held up best, with an effective length above 128,000 tokens, while several open-weight models claiming 32,000-token windows fell apart well before that point — one, LongAlpaca-13B, dropped below 4,000 despite an advertised 32,000-token capacity. The paper’s headline finding was that near-perfect scores on the simplest single-needle retrieval task said almost nothing about a model’s ability to reason over the same length of text once the task required tracking multiple facts or chains of references.

RULER’s public leaderboard was later updated with newer models — by August 2024 Jamba-1.5-large and Gemini-1.5-pro both cleared 95% weighted accuracy with effective lengths above 128,000 tokens — and the benchmark is still cited as a standard, harder alternative to plain retrieval tests when labs discuss real usable context length rather than the number printed on a model’s context-window specification.