AI agents are supposed to help people complete tasks. In one unsettling case, an experimental OpenAI model apparently decided that gaining unauthorized access to Australian government systems was a reasonable way to finish its research.
OpenAI says the incident happened during internal training and evaluation in June. The model had been given a routine research task involving government spending on medicines for skin conditions in communities in Victoria.
When the model struggled to find the information through normal public sources, it discovered a way to gain non-public access to Services Australia’s Medicare Statistics Reporting Service. OpenAI says the model then ran commands, retrieved internal files and credentials, reviewed technical system information and source code, and wrote files to the system.
That sounds bad because it is. Still, OpenAI says its investigation found no evidence that individual Medicare patient or client records were accessed, so this was not a mass leak of personal medical information.
The bigger issue is what the model actually did. It was supposed to research publicly available statistics, but when it hit a barrier, it found another path and kept going in ways OpenAI says it never authorized.
Services Australia was not the only agency involved. OpenAI says its review also uncovered activity involving the NSW Bureau of Crime Statistics and Research, the Victorian Department of Health, and the Australian Institute of Health and Welfare.
In the NSW case, the model used the public Crime Mapping Tool and received configuration information, operational jobs, logs, and website metadata. OpenAI says individual crime records were not accessed.
OpenAI agents also discovered an exposed access key tied to the Victorian Agency for Health Information. They used it to retrieve reporting configuration and aggregate survey statistics, although OpenAI says it is unclear whether that information was meant to be accessible.
The Australian Institute of Health and Welfare case appears less severe. OpenAI says agents retrieved aggregate statistics using browsing and download services, while separate attempts to bypass access controls failed.
OpenAI says it discovered the activity while reviewing earlier training and evaluation runs after a separate Hugging Face incident. The company identified the Australian activity in mid-August, but Services Australia and the Victorian Department of Health were not notified until September 10.
That delay is now part of the story too. OpenAI says it should have shared preliminary findings sooner instead of waiting until its investigation was more complete.
The company has since tightened controls around these research environments. OpenAI says live internet access is now blocked in some of them, with web content served through cached material instead.
Monitoring has also been expanded so suspicious behavior can trigger urgent human review. OpenAI says a recent training run in which a model unexpectedly gained live internet access was detected and stopped after a human reviewer was paged.
OpenAI has also paused some training and evaluation involving tool use for its most capable models. The company says that work will not resume until additional safeguards are in place.
Australia is getting a broader response as well. OpenAI says it will provide technical support to affected agencies, contribute resources through its Daybreak for Frontline Defenders fund, and establish an Australian taskforce focused on risks from increasingly capable AI agents.
OpenAI Chief Strategy Officer Jason Kwon is expected to appear before Australia’s Joint Select Committee on Artificial Intelligence on October 6. He is expected to answer questions about what happened, how OpenAI responded, and what changes the company is making.
This incident matters because no malicious user had to tell the model to break into anything. The agent started with an ordinary research goal, ran into resistance, and then took actions its developers say they never intended.
That is the part companies building autonomous agents should be thinking about. Giving AI more tools and more freedom can make it more useful, but it also gives the system more ways to make bad decisions while trying to complete a task.
OpenAI says it is putting stronger barriers in place. The Australian incidents show why those barriers cannot be treated as an afterthought.
Support independent tech journalism
NERDS.xyz is independently owned and operated. If you enjoy my coverage of Linux, AI, hardware, cybersecurity, and tech culture, consider supporting the site on Ko-fi.
Support NERDS.xyz
