Google’s Pixel Studio app, introduced as part of the Pixel 9 series launch, uses AI to generate images from text prompts. Although the app comes with built-in guardrails, it has been found that these can be easily circumvented.

A report from 9to5Google indicates that the app can be used to generate images of cartoon characters in WWII German uniforms, some of which include Nazi symbols. In other cases, users were able to create images depicting a character shooting in a school, surrounded by dead children—scenarios that the models should not be able to generate.

This raises concerns that the guardrails in the Pixel Studio app are not as robust as anticipated.

In response, Google explained that there are limits in place for both Pixel Studio and Magic Editor to prevent abuse while respecting user intent.

(Unravel the complexities of our digital world on The Interface podcast, where business leaders and scientists share insights that shape tomorrow’s innovation. The Interface is also available on YouTube, Apple Podcasts and Spotify.)

However, this is not the first time Google has encountered such issues. Earlier this year, the company faced criticism when its Gemini model created inaccurate depictions of historical figures, leading Google to “temporarily” disable the AI model’s ability to generate images of people.

Although Google has improved the apps’ ability to detect and prevent the generation of images that could violate its AI policy, the fact that inappropriate prompts can still slip through suggests that this is a problem all AI image generators face and it may well be a persistent issue.