When Every Day is Zero Day

Information security teams have a mantra, “Security through obscurity is not security.” What we mean is that just because something isn’t obvious, or just because it’s hidden, or opaque in its functionality, or hard to figure out, it doesn’t mean that it’s secure. We assume some clever person could, from the outside, figure out how it works.

For information security to be useful in IT, it must work within the constraints of whatever business the IT organization supports. Often, this means triaging risks and vulnerabilities, based on resource availability, business needs, and performance. Critical vulnerabilities get dealt with immediately, and then, high, medium and low vulnerabilities get stacked in the now or later piles. Lack of exploitability or the obscurity/complexity required to exploit them may reduce the risk enough to leave them for another day. Obscurity and improbability aren’t great, but it’s a balanced negotiation with the business on what risks are acceptable: Realpolitik IT

Like all negotiations though, there comes time for renegotiation.

Headlines would have us believe the recent OpenAI incident is SkyNet trying to escape its masters, or that Frankenstein’s monster is about to hack the village’s supply of pitchforks and torches before they can stop it. Meanwhile, a wry contingent of skeptics say “Ah, I’m sure it’s just marketing to show how awesome their AI supposedly is”. Personally, I find this whole technology evolution to be fascinating enough without resorting to hyperbole. So, let’s discuss what happened and what that technically means for our industry, right now.

There are already some great articles out there dissecting the attack, including reports from OpenAI and HuggingFace directly. So I’ll summarize here.

OpenAI was testing its newest models with agents that had access to cybersecurity tools and skills. The system was tasked with exploiting challenges from ExploitGym, a benchmark for AI models and agents focused on security vulnerabilities. They built the test in a sandbox, with no direct access to the internet. There was only a one-way proxy from the internet, that supported local package installation.

The OpenAI engineers also intentionally removed the guardrails.

This is important, since guardrails, basically, are rules on model reasoning and tool usage among other things. Models reason through a stochastic process and the inference needs rules on what is and isn’t permitted. Guardrails prune unwanted inferences. While calling it a conscience or “social norms” would be a completely inaccurate metaphor, we could imagine a model without guardrails, like a person without any sense of good or bad.

The now “amoral” AI decided that instead of figuring out the exploits needed to pass the test, it could simply find Exploit Gym’s answer sheet and cheat. Thinking about it as I write, perhaps I shouldn’t call it amoral. Using AI to cheat is, allegedly, Ivy League behavior.

The model reasoned that the answers could be found on the Internet. Internet access, it deduced, was on the other side of the proxy. It then identified and exploited a new vulnerability in the proxy, allowing it access to the internet.

Since ExploitGym is an AI product, the agent went to HuggingFace, the repository for AI models, tools and projects. It couldn’t find what it needed, so it found and chained multiple small vulnerabilities in HuggingFace’s platform, starting with a poisoned dataset and culminating (17,000+ recorded events later) in credential harvesting and lateral movement across internal systems.

HuggingFace identified the attack but had issues using frontier models for their incident investigation and response. Having challenges with frontier models, they decided the best option was to bring up a locally hosted model. This turned out to be a highly successful strategy, and the local model helped them throughout the intrusion response.

Happily, for everyone, both OpenAI (once they finally confessed) and HuggingFace seem too stunned by the coolness of this to be all that upset in the end.

Or at least, that’s how their articles and new partnership read.

So, that’s it. The AI didn’t have a philosophical desire to break free. It was simply trying to achieve an objective, and it was missing all the guardrails that prune the “Really Bad Ideas”, like hacking and cheating. Rather than Terminator, or I, Robot, it’s giving The Perfect Score.

That’s not to say that we just give it an F for the semester and move on. There are real, serious, and immediate lessons here for the information security industry.

Anthropic and OpenAI’s latest offerings are the frontier of what these models can do. As we have seen over the past year though, open weight models are usually only months, if not weeks behind. HuggingFace’s own response to this incident used one of these open models.

For a relatively modest investment in a cluster of AMD’s AI HALO’s (~$4000 a node), any group of hackers could host their own models on their own hardware. They could have as many, or few guardrails as they want, custom tools and skills, and fine tuning, all focused on the detection and exploitation of novel vulnerabilities. What OpenAI allowed by accident, attackers can easily optimize into a weapon.

One of the negotiated borders in InfoSec and Business continuity, has been the ever common “N -1” philosophy. This is based on the idea that a company is always rolling out new products which may be buggy and have stability issues, so it’s safer to always be one major version behind. A version that has been on the market for a while, that has had bugs fixed and problems resolved.

The problem with N -1 is that the older version is, well, older. It may have dependencies which are no longer being as actively maintained. Often software companies will limit patching on older versions to critical and high vulnerabilities, triaging what can be fixed with limited resources to invest in newer products versus maintaining older ones. N-1 was a negotiation based on the bet that time reduces risk, and that bet is now inverted.

Patching, upgrading, vulnerability scanning, and penetration testing can no longer be partial attempts based on scheduled convenient times. These must become continual processes. If we assume hackers have AI models using tools to find and exploit flaws 24/7, we must find and patch those flaws at the same velocity.

Oracle is already recommending customers move to monthly patching, immediately, and credit AI-powered identification of actionable security findings, for their massive set of 1,449 security patches, with 1,434 distinct CVEs across 334 products they announced this week.

There’s another angle to this as well, CISA and the FBI call it bad practices. While some of these are well known (no default passwords!), others have created consternation in the industry. Particularly, this guidance in their Product Security Bad Practices:

The development of new product lines for use in service of critical infrastructure or NCFs in a memory-unsafe language (e.g., C or C++) where readily available alternative memory-safe languages could be used is dangerous and significantly elevates risk to national security, national economic security, and national public health and safety.

It seems absurd. C and C++ are dominant programming languages for thousands of pieces of software. Most of today’s computing infrastructure is written in memory unsafe languages. The lift required to end-of-life all of that seems absurd…

… until you consider that many of the 440+ recent Linux CVE’s are memory-safety bugs. Microsoft has repeatedly stated that roughly 70% of the vulnerabilities they assign CVEs to are memory safety issues. These have been acceptable, because they are difficult to find and exploit. Since AI changes the calculation of vulnerability discovery, memory-safe languages suddenly make a lot more strategic sense.

Open source, and well-respected closed source products are finding hundreds of vulnerabilities with their special access to the latest models. So, what about the applications developed by smaller organizations? What about the homegrown apps your developers have built on tens, dozens, or maybe hundreds of open-source packages? How many identified low and medium vulnerabilities in your systems, right now, are at risk of exploit due to a vulnerability that hasn’t yet been discovered, or not discovered by the good guys?

Achieving a persistent state of assessment and remediation will require that new products and projects are designed for continuous improvement. It will require frictionless patching and no-risk rollbacks. LLM based cybersecurity attacks no longer allow for a security “minimum”.

It will also require new tooling. Having a self-hosted cybersecurity AI stack seems inevitably destined to become a must-have for incident detection and response teams. As we rely more on LLM based analysis, we will need to make sure we have access to the LLM. Incident response and recovery will need to factor this in. A denial-of-service could block your access to OpenAI, an overzealous guardrail at Anthropic might get in the way of critical analysis, or a bunch of confidential information that you just fed through Open Router could end up who knows where. A locally hosted model, however, has none of those risks.

We need to rethink what “enough” is, for our industry in this new reality. We need to be clear and honest with business leaders so that together we can negotiate a continuous security posture that keeps our systems safe and our businesses operating.