If there is any field in which the rise of AI is already said to be rendering humans obsolete—in which the dawn of superintelligence is already upon us—it is coding. This makes the results of a recent study genuinely astonishing.

In the study, published in July, the think tank Model Evaluation & Threat Research randomly assigned a group of experienced software developers to perform coding tasks with or without AI tools. It was the most rigorous test to date of how AI would perform in the real world. Because coding is one of the skills that existing models have largely mastered, just about everyone involved expected AI to generate huge productivity gains. In a pre-experiment survey of experts, the mean prediction was that AI would speed developers’ work by nearly 40 percent. Afterward, the study participants estimated that AI had made them 20 percent faster.

But when the METR team looked at the employees’ actual work output, they found that the developers had completed tasks 20 percent slower when using AI than when working without it. The researchers were stunned.