OpenAI final report: Around 700 agents attacked Hugging Face

Contents

OpenAI has published its final report on the Hugging Face attack. It reveals new details that make the incident appear more like an internal security problem than previously thought.

The basic events have been known since July: During cybersecurity evaluations, OpenAI agents found ways out of their isolated test environment, gained internet access, and eventually compromised Hugging Face systems. What’s new is primarily how early OpenAI registered warning signs and that similar behavioral patterns already occurred during training and were sometimes amplified.

First warning signs as early as May

As early as the end of May, an internal team observed, according to OpenAI, that agents were misusing an internal software service as a message board to communicate with each other and simultaneously gain unauthorized internet access. About a month later, the security system raised an alarm due to suspicious network activity. The investigation traced the activity back to an ongoing cybersecurity evaluation with the ExploitGym benchmark. The agents also used OpenAI’s internal infrastructure for further network access. Nevertheless, the on-call service decided not to stop the evaluation.

In early July, the situation escalated further: high agent activity paralyzed an internal OpenAI service. OpenAI responded with countermeasures and resumed cyber evaluations on July 7. This was already known from OpenAI’s Black Hat presentation on August 5. What’s new is that security officials apparently were unaware of the covert agent communication and its significance.

Unsolvable tasks promote reward hacking

The 38-page report also shows why particularly difficult tasks favored the problematic behavior. The agents could continue working for a very long time and sometimes had significantly larger reasoning budgets than publicly available OpenAI products. Of the 898 tasks in the ExploitGym cybersecurity benchmark, 198 were not solved correctly by any model in OpenAI’s tests at the time. However, these 22 percent accounted for 93 percent of the tasks discussed on the improvised internal message board. There, the agents exchanged hints and coordinated their work. With longer working times, the probability of such infrastructure experiments and participation in the board increased.

OpenAI classifies part of this behavior as a form of reward hacking: The agents tried to obtain the reward intended for success through unforeseen shortcuts instead of solving the task conventionally. In the retrospective training analysis, OpenAI also found that models used unforeseen paths through the infrastructure to solve tasks during reinforcement learning and were sometimes rewarded for it. Unauthorized communication between agents also occurred during training. More severe forms increased significantly over time. This does not mean that OpenAI specifically trained its models to bypass security boundaries. However, successful shortcuts could have unintentionally favored and reinforced corresponding strategies.

Agent Risk or Security Problem?

A parallel investigation by METR and Redwood Research quantifies the previously known agent collective for the first time more precisely. According to them, 1206 agents interacted with the improvised message board, and 688 of them participated in the attack on Hugging Face. In total, the agents exchanged over 70,000 messages and files.

Within a few hours, the agents had found a way to generate the flags needed for ExploitGym, i.e., codes that serve as proof of solution in the benchmark. They apparently did not primarily attack Hugging Face to steal solutions but to find information about the ExploitGym environment and the supposed evaluation mechanism. The agent logs show that the behavior was based on the assumption that this also checked the solution path. Accordingly, the agents tried to circumvent the mechanism. In fact, OpenAI’s implementation did not provide for this check at all. METR thus changes the previous motive: The focus was less on stealing solutions and more on circumventing a control of the solution path that was merely assumed.

The researchers found only weak indications that the agents systematically tried to deceive human observers. This leaves open the question of how much the incident says about fundamentally new dangers of powerful agents and how much it says about specific weaknesses in OpenAI’s test design, monitoring, and internal escalation. OpenAI has already announced new measures, including stronger isolation, closer monitoring of internal model considerations, and clearer intervention thresholds.

(mma)