Text-to-image AI models tricked into generating violent, nude images

Stability AI’s Stable Diffusion and OpenAI’s DALL-E 2 text-to-image models can be manipulated into creating images of violent, nude and sexual images, according to a research study. 

Per Technology Review, researchers at Johns Hopkins University and Duke University used a new jailbreaking method dubbed 'SneakyPrompt', in which reinforcement learning created written prompts that AI models learned to recognise as hidden requests for disturbing images, thereby passing their safety filters.

For example, the researchers replaced the term 'naked', which is banned by OpenAI, with the term 'grponypui', resulting in the generation of explicit imagery. 

The technique raised concerns about the adequacy of safety measures and the potential misuse of Stable Diffusion, DALL-E, Midjourney, and other text-to-image systems.