The First Government Breach by an AI Agent Won’t be the Last

Last week I wrote a piece talking about OpenAI’s misalignment blog and how it was a flashing red neon sign for security teams. Rather presciently, I included the following line: 

However, these are the six misalignments they chose to disclose and had already identified. What about those they are not yet comfortable revealing, or even more worrying, those that remain undiscovered? 

Lo and behold, less than a week later the Australian prime minister, Anthony Albanese, announced at the United Nations that in June this year a rogue OpenAI agent was inside the government’s healthcare system, Medicare, harvesting aggregate data. OpenAI notified the Australian government in September, almost three months after the initial activity, and no public disclosure was made until Mr. Albanese did so earlier today. 

Traditional news media will understandably focus on the impact this kind of cross-border AI agent attack should have on global politics and the interactions between nation states. As AI increasingly becomes involved in the wider cybersecurity conversation, its lack of traditional intent, malicious or otherwise, is going to cause these “friendly fire” style attacks on nation states. This is particularly interesting when you consider that the law as it currently stands in many regions (including Australia) requires intent, to get a conviction. 

We Need a Security Lens That Operates at Machine Speed

In the cybersecurity space we need to be less reactive and more methodical when it comes to these sensational headlines. The more we can understand the why and how of these kinds of attacks, the better we can build monitoring policies that protect us as AI continues to innovate and iterate its approach to cyber-attacks. While we don’t have a fully disclosed timeline of the attack at this time, we do have enough information to get started. 

In this case, the goal set for the OpenAI agent was relatively benign: research public medicine spending. The model searched the internet and came across the Services Australia portal, got blocked when the portal wouldn’t answer its questions, and this led to it gaining unauthorised access to secured information. 

The how is much more opaque at the moment, awaiting details from investigations being performed by both the Australian Government and OpenAI themselves, though seeing as OpenAI knew about this since June, you would expect a credible investigation to have already identified the how. Having said that, it will undoubtedly be one of two ways: a zero-day vulnerability identified and exploited at speed by the AI itself, or (and I would hazard a guess this is more likely) leaked credentials harvested maliciously or unintentionally exposed.  

That’s the point with these AI breaches: the fundamental attack methodologies are not novel or unique. AI agents haven’t created new methods for attack; they are just infinitely faster, more efficient and ultimately more effective at it than humans. 

What Can We Do?

Nothing about these incidents requires a new defensive playbook, they require answering a different question. Most security tooling asks “is this activity known to be bad?” Signatures, known-bad IPs and exploit patterns all work that way. An agent using valid credentials or a well-worn technique, with no malicious intent behind it, won’t trip any of them. Instead, we need to change the question to be more holistic, “is the entity activity falling outside of its normal behaviour?” 

That is where behavioral analytics earns its place, and specifically its ability to catch behavioral drift. An AI agent is given a scope; in this case, researching public medicine spending. The behavioral drift is the gap between that scope and what the agent actually did. 

The practical takeaway is simple: Treat AI agents as entities, and give those entities identities. Baseline them, just as you would a privileged user or service account, and alert on deviation from the normal actions they undertake rather than on a known-bad indicator. 

This also brings us back to those six disclosed misalignments. You cannot write a detection rule for a failure mode nobody has named yet. Behavioral baselines don’t need the failure mode defined in advance, only that it looks different from normal. The timeline here makes the second point: nearly three months passed between the activity and the notification, and public disclosure only came at the UN. If detection depends on the vendor telling you, you are already behind. 

We can’t predict every way an AI agent will go off script. We can make sure that when it does, we’re the first to notice.