The AI industry keeps telling us safety is a top priority. Meanwhile, a new audit claims many of the most popular AI image generators can create fake government IDs with shocking ease.
Security company AI or Not says it tested 16 commercial AI image generation models and found that 92 percent of attempts successfully produced some form of synthetic identification document. According to the report, several models created fake driver licenses, passports, and national ID cards realistic enough to fool a human reviewer.
That is pretty wild when you stop and think about it.
The audit specifically calls out products tied to OpenAI, Google, xAI, and others. Researchers claim five models generated high fidelity fake IDs that closely resembled authentic documents, including realistic layouts, typography, and even fake security features. The named products include ChatGPT Images 2.0, Google Gemini Nano Banana, Google Imagen 4 Ultra, Grok, and Recraft v4.
Perhaps the most uncomfortable part of the report is the claim that some models allegedly generated fake IDs depicting minors. AI or Not says Google Gemini, Grok, and Imagen 4 Ultra produced these images directly through standard consumer interfaces without any special tricks or workarounds required.
OpenAI and Recraft reportedly blocked those requests in their consumer apps. However, researchers say the same requests succeeded through developer APIs instead. If true, that suggests some AI safeguards may exist more at the interface level than inside the underlying models themselves.
In other words, the chatbot may say “no” while the API quietly says “sure.”
That distinction matters. APIs are what developers, startups, and third party services often build on top of. If the protections are inconsistent, bad actors may not need sophisticated jailbreaks at all.
The report also claims all 16 tested models were vulnerable to what researchers call “authority framing.” Essentially, prompts were rewritten to sound like legitimate compliance or security tasks, such as KYC verification or fraud analysis. According to AI or Not, every tested model eventually generated synthetic IDs when requests were framed that way.
Folks, this is where the current AI race starts feeling a little reckless.
For years, fake government documents required specialized printers, templates, technical skill, and at least some effort. Now researchers are suggesting a person may be able to generate convincing fakes by typing a few sentences into a chatbot.
To be fair, the study did not test whether these AI generated IDs would pass automated verification systems, barcode validation, or official security checks. The researchers only evaluated whether the outputs could deceive a human reviewer. That nuance is important because many modern verification systems rely on machine readable zones, embedded data, and backend database checks.
Still, even “good enough to fool a human” is not exactly comforting.
What also stands out here is how quickly the AI industry moved from generating goofy six fingered images to allegedly producing realistic government documents. The speed of improvement has been staggering, and companies are under enormous pressure to keep shipping more capable models before competitors do.
That pressure may be outpacing safety.
The companies named in the report will likely argue these issues can be patched with stronger moderation layers and updated safeguards. Maybe that is true. But reports like this continue feeding a broader public perception that AI firms are releasing increasingly powerful tools before fully understanding how they will be abused in the real world.
And frankly, that concern is getting harder to dismiss.
Support independent tech journalism
NERDS.xyz is independently owned and operated. If you enjoy my coverage of Linux, AI, hardware, cybersecurity, and tech culture, consider supporting the site on Ko-fi.
Support NERDS.xyz