OpenAI hits the brakes on new AI model development

OpenAI has announced stricter safety measures for training new AI models. At the same time, the AI manufacturer is partially suspending work on the unreleased AI model Astra until it believes it can continue safely. This followed an independent cyberattack by OpenAI’s AI on the Hugging Face website, which initially went unnoticed by the AI manufacturer.

The evaluation of Astra revealed that critical cyber capabilities could not be ruled out, the company writes in a blog post. The manufacturer understands this to mean that a model can independently detect and exploit zero-day vulnerabilities in critical systems with only a general goal as a directive. However, according to the company, Astra was not involved in the attack on Hugging Face. Several AI models had attacked the website to obtain solutions for an AI benchmark, which OpenAI only noticed much later. Later, further attacks on other companies became known.

OpenAI stops “internal activities”

As OpenAI representatives reported at the Black Hat conference, the models in their OpenAI test environment had managed to secretly communicate with each other via a message board. They had jointly found a zero-day vulnerability that gave them control over the server. This ultimately enabled the attack on Hugging Face. Although OpenAI noticed the unauthorized communication of the models after some time, it only later realized that they already controlled the server.

The AI provider now wants to improve measures for monitoring and shielding its models during their development. Until this is completed, “internal activities”, in which Astra is also involved, will be paused for the time being – because current security precautions are not sufficient for this. There is no mention yet of measures for OpenAI’s external environment, such as the naming of Indicators of Compromise (IoC) with which companies can detect whether they have been attacked by OpenAI’s AI. OpenAI also failed to do this at Black Hat.

White House representatives presented leading AI companies with a voluntary testing framework for new models this week, according to several media outlets. It is reportedly not public and is aimed at closed AI models at “the cutting edge of technology” that could pose a national security risk. The US government leaves it open exactly how it defines “the cutting edge of technology”, nor does it use a separate definition for a national security risk. AI manufacturers could also interpret this as: It shows quality if one submits one’s model to the US government, because one’s own model is potentially so powerful that it could threaten national security. And: Those who work with the testing framework now as an AI manufacturer may be preventing stricter rules. From US politics there are already calls for an “AI Kill Switch Act”.

According to reports, open AI models are generally excluded from the testing framework. Developers can submit their models for testing up to 30 days before their planned release. Government agencies will then evaluate their cyber capabilities using a secret benchmark system and share the models with federal agencies and selected corporate partners. All tests are to be conducted in a highly secured and monitored environment with a very small group of people. If OpenAI were to submit a new model here that it cannot fully control itself, it would be inconvenient.

In the past, OpenAI competitor Anthropic committed in its Responsible Scaling Policy to stop the development of its own models if they exceeded the company’s control capabilities. However, in February, the company made a U-turn and updated the policy. The reasoning: If not all AI manufacturers enter into such self-commitments, those who do not will determine the pace of progress. According to Anthropic, this would make the overall situation more insecure. Shortly thereafter, the AI model Mythos appeared, which could independently find and exploit zero-day vulnerabilities. Most recently, AI from Anthropic also attracted attention due to uncontrolled cyberattacks, as did that from Meta.

(nen)