OpenAI publishes 'Advancing red teaming with people and AI'

OpenAI published two papers describing how it tests models for vulnerabilities before release, one on its approach to external human red-teaming and one introducing an automated method.

The first, a white paper on external red-teaming, set out how the company selects testers with relevant expertise — cybersecurity, natural science, regional politics — decides what access to give them, briefs them, and synthesises their findings into safety evaluations. OpenAI said the approach had recently been used to prepare the o1 model family for public release, and cited earlier campaigns, including testing of DALL-E 2 in 2022, as precedent.

The second paper addressed a limitation of automated red-teaming: that models trained to find attacks tend to converge on a narrow set of known strategies rather than discovering new ones. OpenAI’s proposed fix, multi-step reinforcement learning with auto-generated rewards, used a more capable model to generate a diverse set of attacker goals and then trained a separate model to pursue them, rewarding both success and variety of approach.

The company was candid about limits on both methods: red-teaming, human or automated, only captures risks present in a system at a specific moment, and disclosing found vulnerabilities carries some risk of handing attackers a map. The publication set out a methodology OpenAI would cite repeatedly in later system cards, including for Sora, as its standard account of how pre-release testing was conducted.