InfiniteBench

InfiniteBench was built to fill a gap that opened as context windows raced past what most existing tests measured: in early 2024, when it was released, public long-context benchmarks mostly topped out around 10,000 tokens, while frontier models were beginning to advertise context windows in the hundreds of thousands. Researchers associated with Tsinghua University’s OpenBMB project built 12 tasks — spanning simple key and passage retrieval, English and Chinese comprehension, code debugging and execution, and mathematics — with average example lengths above 100,000 tokens, and deliberately included tasks that could not be solved by retrieving just a few relevant passages.

Results at launch split sharply by task type. On the simplest retrieval tasks, passkey and number lookup, both GPT-4 and the open-weight Yi-34B-200K scored a perfect 100%, showing that basic long-range retrieval was already close to solved for capable models even at these lengths. On harder tasks that required using information rather than just finding it — such as locating and combining specific numbers scattered through the document in the “Math.Find” task — scores dropped sharply and separated models clearly, with GPT-4 at 60% against Yi-34B-200K’s 25.71%.

That split became InfiniteBench’s lasting contribution: it showed early that “can a model retrieve a fact from very far away” and “can a model reason over a very long input” were different capabilities that saturated at different rates, a distinction that later long-context benchmarks such as RULER and LongBench v2 built on more systematically.