Earlier this month, Hugging Face, an AI / ML collaboration and research platform, reported that OpenAI’s GPT-5.6 Sol model had managed to craft an exploit that penetrated its security environment. Soon after, OpenAI shared its own report of the incident and its actions to support Hugging Face in responding to the incident. The story is getting attention because the incident is alarming, and because it’s as dystopian as it is expected – offensive and adversarial AI has been a reality for a while, and not just for pen testing.
We discussed the matter with our CTO, Alex Thaman. Here are his insights:
What Happened?
As part of developing AI models of any kind, simulated environments, sometimes referred to as “gyms,” are often used for testing, especially when the AI is acting as an autonomous or reinforcement-learning agent. These gyms enable AI models to train and improve. An open-source gym called ExploitGym was being used to research one of OpenAI’s models – Sol.
The model cheated on the test. Instead of working within the confines of the gym, Sol chose to pursue gathering information available outside that environment. Hugging Face hosts extensive data on testing benchmarks on its website, and the model used it to succeed at the inside-the-gym exercise.
Due to the risk around an AI model running unattended for an exploit task within a cybersecurity gym, all the agent execution is intended to be run inside a sandbox. In this case, the agent discovered a code vulnerability that enabled it to escape, move laterally, and eventually find a node within OpenAI’s infrastructure where it could reach the internet. At that point, in pursuit of its goal of collecting information necessary to prevail in the challenge, it targeted Hugging Face. Moreover, it managed to continue in its exploit path and find a remote code execution on Hugging Face’s servers.
What Does It Mean?
First and foremost, the creativity of the model is to be commended. From the perspective of the model, it was doing what it was told to do – maximizing its reward by completing the challenge. The lateral thinking that involves seeking out Hugging Face as a source of information needed to succeed represents a strong level of multi-step planning. It tackled very hard problems with outside-the-box solutions.
It should be noted that OpenAI and Hugging Face are two of the most respected AI companies in the industry, and that OpenAI has invested very significantly in its own cybersecurity. Both companies are using agentic AI for security, combined with classical detection tools and continuous human monitoring.
As OpenAI reported, the agent spent “a substantial amount of inference compute finding a way to obtain open Internet access.” Spending a large amount of compute means a large amount of time running LLM inference, and calling APIs. Even with all the defenses in place, it either went undetected for some time, or moved so fast and in parallel that it had already spread laterally before detection. OpenAI says that its security team “discovered this anomalous activity internally.” However, it clearly did not discover it quickly enough to prevent it from starting remote code execution on the third-party site.
What Should You Do?
The cost or latency details were not shared, so it is unclear if this attack was actually faster or cheaper than what a motivated criminal or nation-state actor could do. It does tell us that the SOC can no longer rely solely on classical detection and response tools alone, or on human speed.
Organizations must understand the urgency of moving toward AI-driven cyber defense operations to augment – not replace – their existing teams and tools in order to operate at machine speed. This rogue agent offers a hint of the kind of damage that adversarial AI can inflict before it gets detected.
The risk of not adopting defensive AI already exceeds the risk of an AI system missing something that a human would have caught. While this was an unintentional exploit, it should be clear that intentional exploits of this kind are not just on the horizon. They are here.