DeepMind's XLand agents generalise across millions of open-ended games

DeepMind reported that reinforcement-learning agents trained inside XLand, a procedurally generated 3D environment posing millions of variations on games such as hide-and-seek, tag and capture-the-flag, could perform competently on games they had never encountered during training. Agents in the final training generation had faced 3.4 million unique tasks across roughly 4,000 procedurally generated worlds and around 700,000 distinct games in total, accumulating 200 billion training steps.

The training method was iterative rather than a single run: an automated curriculum selected tasks near the frontier of what the current generation of agents could already do, and each successive generation was distilled from the strongest performers of the last. DeepMind reported that the resulting agents succeeded at nearly every procedurally generated evaluation task they were tried on, including ones built specifically to be unlike anything in training — behaviours such as stacking movable blocks into a ramp to reach a goal emerged from this process rather than being hand-specified.

The result was offered as evidence that breadth and diversity of training tasks, combined with a curriculum that kept pushing agents toward their current limits, could substitute for hand-designed reward shaping or task-specific architectures — an argument about scale and generalisation, made in the language of reinforcement learning rather than of the large language models that were coming to dominate the field’s attention at the same time.