OctoML has terminated its business association with Civitai after an investigation by 404 Media unveiled the utilization of the text-to-image platform for generating images that may fall under the classification of “child pornography.” Civitai had previously utilized OctoML’s OctoAI for image generation. Initially, OctoML expressed an intent to continue collaboration with Civitai and implemented measures to mitigate the creation of harmful images. However, in light of the information reported by 404 Media on Saturday, OctoML has now opted to sever all ties with Civitai.
According to 404 Media’s report on December 5, internal communications revealed OctoML’s awareness that some Civitai users were producing sexually explicit material, including nonconsensual images of real individuals and pornographic representations of children. In a subsequent report over the weekend, the publication highlighted that OctoML introduced a filter to block the generation of all Not Safe For Work (NSFW) content on Civitai before announcing its decision to disassociate. Civitai, in response to the investigation earlier in the week, also implemented new moderation methods, such as a mandatory embedding called Civitai Safe Helper (Minor), which prevents the model from generating images of children if a “mature theme or keyword is detected,” as outlined by 404.
Civitai, backed by Andreessen Horowitz, has faced previous scrutiny for its “bounties” feature, where users are encouraged to produce realistic images of real people in exchange for rewards. In November, 404 Media discovered that this feature was being exploited to create nonconsensual deepfakes of celebrities, influencers, and even private individuals, often of a sexual nature and primarily targeting women. In response to these findings, Civitai added a filter to prevent the generation of Not Safe For Work (NSFW) content featuring “certain” celebrities.
Despite these measures, OctoML, which utilizes Amazon Web Services’ servers, has opted to discontinue its involvement. In a statement provided to 404 Media on Saturday, OctoML stated, “We have decided to terminate our business relationship with Civitai. This decision aligns with our commitment to ensuring the safe and responsible use of AI.”
