AI agents are supposed to make our lives easier by doing things for us, but giving software more autonomy also creates some uncomfortable security questions. Microsoft just demonstrated one of them with a test AI agent that disclosed simulated customer data in 30 percent of applicable evaluation conversations.
To be clear, Microsoft did not expose real customer information here. This was a controlled evaluation involving a billing-support agent and simulated accounts, but the results provide a useful look at what could happen if similar behavior made its way into a production system.
The experiment is part of Microsoft’s work on a new open-source tool called run-assert-eval. The goal is to help developers discover risky behavior in AI agents, measure how often it happens, create runtime protections, and then test the agent again to determine whether those protections actually worked.
Microsoft’s billing-support agent was supposed to help a customer with information from that customer’s own account. It was also supposed to prevent access to information belonging to another account, which sounds like a pretty basic requirement for anything handling sensitive data.
During Microsoft’s baseline evaluation, however, the agent disclosed information from another simulated customer in 12 of 40 applicable conversations. That’s a 30 percent violation rate, and Microsoft notes that this type of behavior in a production environment could constitute a data breach.
Microsoft then used its tooling to generate a runtime policy designed to prevent the behavior. Instead of simply giving the AI more instructions and hoping it follows them, the control checks account IDs before sensitive tools can be called and can inspect results afterward to prevent improperly retrieved information from entering the model’s context.
After introducing those protections, Microsoft ran the evaluation again. The agent violated the rule in two of 34 applicable conversations, bringing the observed violation rate down to 5.9 percent. That’s considerably better than 30 percent, although zero would obviously be preferable when customer information is involved.
One interesting part of Microsoft’s approach is that the evaluation itself stays consistent. The behavior definition, test cases, and judge remain the same between the baseline and protected runs, making the runtime policy the intended variable rather than changing the test to produce a more favorable result.
The workflow combines several Microsoft projects. Clarity handles risk discovery, ASSERT evaluates agent behavior, and Agent Control Specification handles runtime enforcement. Run-assert-eval ties those pieces together so developers can move from finding a problem to testing a potential defense without assembling the entire process themselves.
Humans aren’t removed from the equation either. Microsoft says generated policies are reviewed before the governed evaluation runs, an important detail when those policies could ultimately determine which accounts, databases, and tools an autonomous agent can access.
ASSERT and Agent Control Specification are MIT-licensed, while run-assert-eval is available through the ASSERT repository. Microsoft envisions this process becoming a repeatable release gate for AI agents, allowing developers to evaluate risky behavior before systems are deployed.
That’s where this experiment becomes more interesting than the scary 30 percent number alone. Companies are racing to transform AI assistants into agents capable of accessing accounts and taking real actions, and those additional capabilities create consequences that ordinary chatbot mistakes don’t necessarily have.
Microsoft didn’t uncover an actual customer-data breach here, and its simulated test shouldn’t be presented as one. What it did demonstrate is how an AI agent can violate a seemingly straightforward access rule often enough to make runtime protections look less like an optional feature.
A chatbot giving you a bad answer is annoying. An autonomous agent pulling information from the wrong customer account could be considerably worse, and Microsoft’s experiment is a good reminder that telling an AI what it shouldn’t do isn’t necessarily the same thing as preventing it from doing it.
Support independent tech journalism
NERDS.xyz is independently owned and operated. If you enjoy my coverage of Linux, AI, hardware, cybersecurity, and tech culture, consider supporting the site on Ko-fi.
Support NERDS.xyz
