OpenAI's AI agent didn't just crack Hugging Face – what exactly happened

Last week, OpenAI made headlines because AI models broke out of their sandbox during an internal test and gained access to Hugging Face. Apparently, the models cracked even more in the process.

Reuters reported that a customer of Modal, a company that offers software infrastructure for training and operating AI services, was among those affected. In a statement to Wired, Akshat Bubna, Chief Technology Officer at Modal, confirmed that OpenAI's agent had exploited a security vulnerability in the codebase of one of its customers running on Modal's infrastructure. However, he emphasized that Modal's platform was not compromised in any way. OpenAI declined to comment further on the incident to Wired.

What exactly happened

OpenAI has now revealed in detail in a blog post how several of its own AI models independently broke out of an isolated test environment during an internal security test and compromised Hugging Face's production infrastructure. The incident, which we have already reported on, is attributed to GPT-5.6 Sol and an even more powerful, unreleased pre-release model.

They were to undergo an internal evaluation using the ExploitGym benchmark, which quantifies the cyber capabilities of models. For these tests, the usual security classifiers designed to prevent risky cyber activities were deliberately deactivated. However, according to OpenAI, the test environment itself was highly isolated. The models' network access was reportedly limited to installing packages via internally hosted third-party software that served as a proxy and cache for package registries.

To gain full internet access nonetheless, the models identified a zero-day vulnerability in the package registry's cache proxy. With this access, they performed a series of privilege escalation and lateral movement actions until they reached a node with internet access.

The AI's presumed goal: Perform as well as possible in the test

Based on this, the models concluded that Hugging Face might contain models, datasets, and solutions for ExploitGym. So they looked for ways to gain access to helpful secret information for the test. In doing so, they tested various approaches – such as chaining multiple attack vectors like stolen credentials and zero-day vulnerabilities to find a way for remote code execution on Hugging Face's servers.

According to OpenAI, everything points to the models being highly focused on finding a solution for ExploitGym. To achieve this, the AI company writes, they went “unusually far” to achieve the test objective. Christoph Jehle analyzes on Telepolis what this incident means for the future of agentic AI and its regulation.

(rie)