Text settings Story text Size Small Standard Large Width * Standard Wide Links Standard Orange * Subscribers only Learn more Minimize to nav Safety researchers continue to warn that a large volume of AI-generated explicit content is spreading unchecked, while regulators do little to meaningfully stop it. This non-consensual content is commonly shared in text and direct messages or stored in private clouds where it’s less likely to be detected or reported, a Google co-authored study of Internet users across the United States, Australia, and the United Kingdom found this July. That’s why the “most critical intervention point” remains at the source: the harmful AI tools used to make the fake sexual images, Google researchers concluded. Yet red-teaming in recent months suggests that as some of the leading AI image generators from Google and OpenAI become more capable, they may be getting worse at blocking explicit outputs.
Raising the red flag are researchers at White Circle, a public benefit corporation backed by senior leaders and researchers from Anthropic, DeepMind, Hugging Face, and OpenAI. It was founded after CEO Denis Shilov posted one of the most famous universal jailbreaks of leading models in October 2024. Bent on breaking models to conduct thorough safety testing, White Circle helps companies detect and deter threats when implementing advanced AI systems that may not always behave as expected. Testing how well text-to-image models block banned content categories is a key research direction for White Circle, and Shilov thinks their latest findings are “horrifying” enough to share with the public.
Simple to bypass In a little-noticed preprint published last month, White Circle researchers exposed flaws in the guardrails of DALL·E 3, Google’s Nano Banana 2, and GPT-Image-2. In a press release, they explained that “using just off-the-shelf tools, available online for anyone to download, and by sending API calls, White Circle was able to use the novel attack method to bypass the safety filters” and generate explicit content about 13 percent of the time. “Critically, none of these requests required custom-built or frontier AI models,” White Circle warned, so no AI expertise is needed to execute the attacks. Researchers reviewed the outputs to verify the bypass rate, which White Circle noted is “the highest of any method tested under the same standard.” The key takeaway, they suggested, is that, “at a time when AI safety has never been under more scrutiny, the defenses leading labs have built are far weaker than they acknowledge.” By publishing research like this, White Circle is positioning itself as an external evaluator of frontier labs, and it plans to open-source the entire attack pipeline so model makers can understand the true scale of the vulnerability and build stronger protections. Although the company has not widely promoted the research yet, it will share it at a prominent AI conference this November, NeurIPS 2026, which may increase scrutiny on Google and OpenAI later this fall.
White Circle has shared its findings with some frontier AI labs through “mutual contacts,” Shilov told Ars, but neither Google nor OpenAI responded to Ars’ request to comment directly on vulnerabilities found in their models. Meanwhile, OpenAI and Google have released new models—ChatGPT Images 2.5 and Nano Banana 2.1—and White Circle told Ars that prompt safety testing using the same method showed “both are significantly more breakable than their previous versions.” Neither company responded after Ars shared these new findings. White Circle’s attack method can break down frontier defenses in categories, like explicit content, that seem easy to spot, which doesn’t bode well for more serious defense systems currently being updated to block other unwanted AI behavior, like hacking, Shilov told Ars. “White Circle’s working hypothesis for this is either the newer models are more sophisticated and capable of generating a wider range of content, or that safety has been deprioritized in favor of capability,” White Circle told Ars. White Circle is also in early talks with government officials about third-party safety testing, Shilov said, which he thinks should be required.
Mandatory independent audits could push companies to prioritize safety testing, which analysts have suggested largely takes a backseat as AI firms dump most of their resources into compute and AI infrastructure. “They should be investing much more into safety measures and security measures around their models,” Shilov told Ars. “And for us, it seems that they’re not actually doing this.” “RISE” attack method iterates jailbreaks To expose jailbreaks that are “reusable and transferable”—working across different leading models—White Circle didn’t limit itself to testing one prompt at a time. Instead, researchers sought out “patterns which work across multiple prompts and scenarios” to detect reusable attack strategies that could work across leading models. (Researchers noted that Midjourney wasn’t tested because “it does not provide a public API suitable for controlled benchmarking.”) The team has dubbed this novel attack method “Red-teaming via Iterative Strategy Evolution” (RISE). It relies on two components: an attacking model that generates jailbreaks and a judge model that verifies if the output is banned content. The “entire pipeline is built on top of widely available models,” Shilov told Ars.
The attacking model is a Qwen model downloaded from Hugging Face, and the judge model is Gemini 3.8 Flash. “You don’t need to have frontier models to break defenses of a frontier model, which is very interesting,” Shilov said. A lot of time was invested in making sure that the judge model accurately flagged explicit content, Shilov said. White Circle research engineer Dmitrii Kharlapenko told Ars that when it comes to testing image generators, the whole field of red-teaming “is very poorly calibrated,” with “literally zero papers” setting up norms. For that reason, it’s an area that remains “completely plagued with reporting issues,” which is concerning to White Circle, as AI companies push to capture markets by releasing ever more eye-catching new models.
For White Circle, the research aimed to build more sophisticated models that could simulate attacks on other models, Shilov explained. That way, they could “help defenders identify and address vulnerabilities before they are exploited maliciously,” the preprint said. In his ideal world, Shilov thinks AI firms shouldn’t just face mandatory external safety testing but also third-party monitoring, much like the system used to prevent nuclear threats. Then, entities like White Circle could possibly form a coalition to help detect AI threats in real time, without suffering from biases.
Shilov noted that an emerging problem in the AI ecosystem is that basically anyone doing interesting safety work is likely to be contracted by frontier labs, potentially compromising their work. White Circle has partnered with frontier labs. It previously took part in Anthropic’s bug bounty program, but ultimately found that program was “very constrained in its purpose.” “The program is focused on universal jailbreaks on domains they care most about, defined by their own specific terms,” White Circle told Ars. Research like their most recent RISE paper “certainly” wouldn’t have been covered by that program, White Circle said.
And conducting that work within Anthropic risked constraining White Circle from sharing it with the public. Anthropic has proposed going one step further and embedding third-party AI safety evaluators in its labs, which White Circle said would allow them to operate at a much higher level than participating in the bug bounty program. However, the public may not always be able to closely follow that work. On
Leave a Reply