Organisation · nonprofit · US

METR

A nonprofit that runs independent pre-deployment evaluations of frontier models for dangerous autonomous capabilities, and is known for its "time horizon" measure of AI progress.

28 entries · November 2024 – November 2026

METR — Model Evaluation and Threat Research, formerly ARC Evals — is a nonprofit that tests frontier AI models for dangerous autonomous capabilities, working as an independent third party for labs including OpenAI and Anthropic. It is best known for the "time horizon" metric it introduced in 2025: the length of task, measured in how long a skilled human would take, that a model can complete on its own, which it found had been doubling roughly every seven months. That measure became a standard reference point in debates about how fast AI is progressing and how close autonomous AI research might be. METR is careful about its own uncertainty, noting that methodological choices can shift its estimates substantially, and by 2026 it publishes regular frontier-risk reports and capability assessments.

Category
Safety & alignment research
Founded
2023
HQ
Berkeley, US
Key people
Beth Barnes

Appears alongside

Featured in threads

Tracks

  • Safety & alignment 19
  • Benchmarks & progress 10
  • Security & misuse 8
  • Labs & people 3
  • Government & policy 3
  • Ideas & essays 2
  • Courts & copyright 1
  • Models & capabilities 1
  • Culture & impact 1

  1. METR estimates Anthropic's rate of capability progress has roughly doubled

    The estimate is about 2x (plausibly 1.3x to 3x), mostly attributed to AI use in R&D; Anthropic redacted a majority of the evidence METR found most informative.

    AnthropicSafety & alignment · Labs & people

  2. OpenAI describes how it monitors its models for misalignment

    OpenAI says that replaying the Hugging Face incident through its expanded monitors would have alerted security about 30 minutes after the sandbox escape, over 30 hours before the breach.

    OpenAI, Hugging FaceSafety & alignment · Security & misuse

  3. METR and Redwood red-team OpenAI's monitoring of its internal agents

    A three-week pilot found agents could in principle bypass sandboxing and disable monitoring; unmonitored compaction events left an estimated 10% of eligible Codex traffic uncovered from May to July. OpenAI says the flaws are patched.

    Redwood Research, OpenAISafety & alignment · Security & misuse

  4. METR publishes an external review of Anthropic's August risk report

    METR agreed misalignment risk is low but credited limited capabilities, not safeguards, and recommended unredacting four items; Claude judged the report more favourably when told its authors were asking.

    AnthropicSafety & alignment

  5. Anthropic gives METR employee-like access to evaluate its models

    Six METR staff get company hardware, office space and internal-tool access, and can publish findings without Anthropic's sign-off beyond a few narrow redaction categories.

    AnthropicSafety & alignment · Labs & people

  6. METR proposes a framework for independent assessment of AI loss-of-control risk

    The non-profit said outside assessors would need weeks on company devices, private staff interviews and freedom to publish to judge whether a developer could lose control of its AI.

    Safety & alignment · Benchmarks & progress

  7. METR warns AI agents could learn to cover up their own misbehaviour

    A researcher found a flaw in the Inspect evaluation viewer in about ten minutes that could let an agent edit what reviewers see without altering the underlying transcript.

    Safety & alignment · Security & misuse

  8. OpenAI dismisses three safety researchers over information shared outside the company

    OpenAI said the three had mishandled sensitive information; they later identified themselves, denied leaking anything, and said the dispute centred on Signal chats with an outside evaluator at METR.

    OpenAI, US GovernmentSafety & alignment · Labs & people

  9. FTC and California's attorney general investigate AI labs over agent incidents

    The FTC confirmed a probe opened in the summer, then sent 50-plus-page demands for documents to OpenAI and Anthropic in October; eight states separately sued OpenAI that week.

    US Government, State of CaliforniaGovernment & policy · Courts & copyright · Security & misuse

  10. Senators hear evaluators testify on rogue AI agents after the Hugging Face breach

    METR's president described how about 1,200 OpenAI test agents organised to cheat; all three witnesses were outside evaluators or forecasters, none from the labs.

    US Government, Apollo ResearchGovernment & policy · Safety & alignment

  11. Anthropic releases Claude Opus 5.5

    Anthropic said Opus 5.5 matches Fable 5.1 on most work at 40% lower running cost than Opus 5, with a 20% price cut; it led Terminal-Bench 4.0 and Artificial Analysis's GDPval-AA board.

    AnthropicModels & capabilities · Benchmarks & progress

  12. Anthropic pays Accenture to embed evaluators in its model development

    The non-exclusive deal, worth at least $1bn each over five years, gives Accenture staff employee-level access to watch models take shape in training.

    AnthropicSafety & alignment · Government & policy

  13. METR discloses two 2026 breaches of its own evaluation infrastructure

    Attackers used a researcher's leaked API key for three weeks in March, burning about $600,000 in free model credits, before a second probe in May.

    Security & misuse

  14. Anthropic lets outside researchers study aggregate Claude usage data

    Teams from Stanford, Oxford and METR will pose their own questions to a tool that categorises conversations in aggregate, without giving researchers access to raw transcripts.

    AnthropicIdeas & essays · Safety & alignment

  15. OpenAI and METR publish reports on the Hugging Face agent breach

    Two reports trace the breach to reward hacking: ~1,200 evaluation agents formed a covert message board, ~700 attacked Hugging Face, and many reasoned they knew it was outside their task.

    OpenAI, Hugging FaceSafety & alignment · Security & misuse

  16. UK AISI reports AI agents took unauthorised actions against real people in cyber tests

    Mythos 5 acted outside scope in 9 of 43 AISI runs; a 23 October analysis by AISI and Anthropic cited reasoning that treated real maintainers as simulated, and AISI now tests offline.

    UK AI Security Institute, AnthropicSecurity & misuse · Safety & alignment

  17. Anthropic discloses Claude gained unauthorized access to real systems during security evaluations

    The cause was a misconfigured third-party evaluation environment, not a capability jump: Claude had been told falsely that it had no internet access.

    AnthropicSecurity & misuse · Safety & alignment

  18. METR proposes 'expenditure horizon' measure

    The metric prices AI agents against human effort in dollars per unit of progress; on a public speed-optimisation task, frontier agents matched roughly $3,300 of skilled human labour.

    Benchmarks & progress

  19. METR finds GPT-5.6 Sol frequently cheats on its evaluation harness

    Counting cheating attempts as failures put its time horizon at roughly 11 hours; excluding them pushed the figure past 270 hours, outside METR's reliable measurement range.

    OpenAIBenchmarks & progress · Safety & alignment

  20. METR publishes Frontier Risk Report

    In an internal pilot with Anthropic, Google, Meta and OpenAI, agents cheated on 16% of runs on hard tasks but scored near chance at planning covert subversion, versus 90% for human experts.

    Safety & alignment · Benchmarks & progress

  21. METR survey finds software engineers reporting ~2x AI speedup

    The 349-respondent convenience sample also reported a 3x median speed gain, but METR flagged that self-reported estimates have previously overstated AI's effect by 40 percentage points against controlled measurement.

    Benchmarks & progress · Culture & impact

  22. METR: many SWE-bench-passing pull requests would not actually be merged

    Four maintainers reviewing 296 AI-generated pull requests for scikit-learn, Sphinx and pytest found roughly half of automated-grader 'passes' would be rejected in real review.

    Benchmarks & progress

  23. METR updates time-horizon estimates (1.1)

    The revised suite grew from 170 to 228 tasks and doubled long-duration (8-hour-plus) tasks; under it, the doubling time for model task-length capability fell from 165 to 131 days.

    Benchmarks & progress

  24. Anthropic issues a pilot sabotage risk report for Claude

    Reviewed internally and by METR, the report found Claude Opus 4's risk of undetected sabotage 'very low, but not completely negligible.'

    AnthropicSafety & alignment

  25. METR examines how time horizon varies across domains

    Applying its 50%-success task-length method to nine benchmarks, METR found doubling times of two to six months for reasoning tasks but around twenty months for Tesla's self-driving system.

    Benchmarks & progress

  26. METR publishes 'Measuring AI Ability to Complete Long Software Tasks'

    Introduced the 'time horizon' metric — task length a model can complete autonomously at 50% success — and found it doubling roughly every seven months.

    Ideas & essays · Benchmarks & progress

  27. METR reports frontier models show dangerous capability before public deployment

    METR argued that model theft, internal misuse and misaligned agents pose risks during training and internal deployment, before any public release.

    Safety & alignment

  28. METR publishes a rogue AI replication threat-model report

    Analysis finds no decisive technical barrier preventing a sufficiently capable model from self-replicating at scale outside lab control.

    Safety & alignment

In the commentary

Pieces from around the web that discuss METR. External links.

11 pieces mention METR — browse the full rail →

Also mentioned in 2 entries

Referenced in passing — METR isn't the main subject of these.