GraphWalks

Long-context benchmarks often test retrieval: hide one fact in a long document and ask for it back. OpenAI released GraphWalks with GPT-4.1 in April 2025 to test something harder. The whole context is a graph, written as hundreds of thousands of edges between random names, and the model has to follow paths through it — every node exactly three steps from a given start, for instance. OpenAI said the task “cannot be solved sequentially”: the model must connect edges scattered across the whole input.

It became a standard long-context entry in both OpenAI’s and Anthropic’s launch tables through mid-2026, usually reported separately for contexts around 256,000 tokens and around one million. Scores at a million tokens stayed well below those at 256,000. Anthropic’s Opus 4.6 card also found errors in the published answers, which OpenAI corrected in February 2026. Because each lab runs its own version with its own prompt and grader, figures from different labs are only loosely comparable.

Both labs dropped it from their later launches. Google kept using it: its Gemini 4 Carbon table reported 96.2% on the 256k–1M split, against 66.8% for Claude Opus 5.5 and 71.8% for GPT-6 Astra. Google ran all of those models itself, so the comparison is internally consistent but has no independent check.