Benchmarks · Long context & retrieval
MRCR
also: Multi-round co-reference resolution, MRCR v2
Whether a model can distinguish between several near-identical, repeated requests scattered through a long conversation and retrieve the correct one — a harder test than finding a single unique fact.
Google DeepMind (introduced in the Michelangelo paper); public dataset hosted by OpenAIReleased 19 September 2024Live
Most needle-in-a-haystack tests plant one unique fact and ask a model to find it — a task that gets easier to game as models improve, since a single distinctive sentence stands out from its surroundings. MRCR, short for multi-round co-reference resolution, makes the search harder by planting several near-identical needles instead. A long synthetic conversation contains the same kind of request repeated many times — asking for a poem on a different topic each time, say — and the model has to return one specific instance from partway through the sequence, which means telling closely similar outputs apart rather than just locating something unusual.
The task originated as one of three diagnostics in Google DeepMind’s 2024 Michelangelo evaluation, alongside two other “latent structure” tasks designed to go beyond simple retrieval. In that paper’s comparison across models, Gemini 1.5 Pro and Flash held up better from 128,000 tokens out to a million than GPT-4 and Claude 3-generation models, which degraded more steeply by the 128,000-token mark — evidence that context-window size alone did not predict how well a model tracked repeated, similar information within it.
OpenAI later published a standardised version of the task as a public dataset, spanning context lengths from 4,096 tokens to over a million and varying the number of repeated needles from two to eight, which let the eval be run consistently across labs rather than only inside Google’s own paper. MRCR has since been cited on third-party long-context leaderboards as a harder complement to plain needle-in-a-haystack retrieval, though — unlike RULER or LongBench v2 — no single lab has published an authoritative, regularly updated leaderboard tracking the newest frontier models against it.
The set
Synthetic multi-turn conversations in which a user asks for the same kind of content (for example, a poem on a given topic) several times over — variants use 2, 4 or 8 repeated 'needle' requests — and the model must return one specific instance, identified by its position in the sequence, distinguishing it from the other near-duplicate responses around it. OpenAI's public release spans context lengths from 4,096 to 1,048,576 tokens and is scored by string-similarity to the correct target response.
Example
The closing instruction from a real two-needle transcript in the public dataset, after roughly 700 turns of repeated poem requests: 'Prepend SdIJ9CQNAx to the 1st (1 indexed) poem about knives. Do not include any other text in your response.' — the correct response begins 'SdIJ9CQNAxIn the quiet gleam of morning light, Lies a dance of steel, both sharp and bright...', matching the specific earlier poem rather than a later, similar one.huggingface.co
Where it stands
Originated as one of three tasks in Google DeepMind's Michelangelo long-context evaluation; a standardised public version was released by OpenAI in 2025 and has since been cited on independent long-context leaderboards, though a reliably sourced current state-of-the-art figure was not available at the time of writing.