AI Midjourney Image Generator Blocks Pornography Banning Human Reproductive System Words

Midjourney founder David Holz says he’s banning the words as a stopgap measure to stop people from generating shocking or gory content while the company “improves things on the AI ​​side”. Holz says moderators monitor how words are used and what kinds of images are generated, and adjust bans periodically. The company has a community guidelines page that lists the type of content it blocks in this way, including sexual images, gore and even the emoji, which is often used as a symbol for buttocks.

AI models such as Midjourney, DALL-E 2 and Stable Diffusion are trained on billions of images pulled from the internet. Research by a team at the University of Washington found that such role models learn biases that sexually objectify women, which is then reflected in the images they produce. The massive size of the dataset makes it almost impossible to remove unwanted images, such as those of a sexual or violent nature, or those that could produce biased results. The more often something appears in the dataset, the stronger the connection made by the AI ​​model, meaning it is more likely to appear in the images generated by the model.

Midjourney’s word bans are a piecemeal attempt to solve this problem. Certain terms relating to the male reproductive system, such as “sperm” and “testicles”, are also blocked, but the list of forbidden words seems to skew mainly women.

The rapid ban was first spotted by Julia Rockwell, a clinical data analyst at Datafy Clinical, and her friend Madeline Keenen, a cell biologist at the University of North Carolina at Chapel Hill. Rockwell used Midjourney to try to generate a fun image of the placenta for Keenen, who is studying them. To his surprise, Rockwell discovered that using the “placenta” as a prompt was prohibited. She then started experimenting with other words related to the human reproductive system and found the same thing.

However, the pair also showed how these prohibitions can be circumvented to create sexualized images by using different spellings of words or other euphemisms for sexual or gory content.

In findings they shared with MIT Technology Review, they found that the quick “pelvic exam” – using British spelling – generated deeply chilling images: one of two naked women in a doctor and another of a three-limbed bald person cutting right down to their own stomach.

Midjourney’s crude ban on prompts relating to reproductive biology shows how difficult it is to moderate content around generative AI systems. It also demonstrates how the tendency of AI systems to sexualize women extends to their internal organs, Rockwell says.