CivitAI generates synthetic 'child pornography' images

Open source image generator Civitai could be used to make images that ‘could be categorized as child pornography,’ or child sexual abuse material (CSAM), according to a media investigation.

Internal communications at text-to-image platform CivitAI cloud computing supplier OctoML shown to 404 Media revealed that it was aware that some CivitAI users had been creating sexually explicit material, including nonconsensual images of real people and pornographic depictions of children. CivitAI had been using OctoML's OctoAI for image generation.

OctoML responded to 404 Media's report by rolling out a filter to block the generation of NSFW content on CivitAI, before cutting ties with the company. 'We have decided to terminate our business relationship with CivitAI. This decision aligns with our commitment to ensuring the safe and responsible use of AI,' the company told 404 Media.

CivitAI founder Justin Maier later told Venture Beat that he had been aware that some people were making NSFW content, but that he tolerated it as they were helping to train the company's models.