DALL-E image generator

DALL-E is a software programme that automatically generates images from natural-language text descriptions (or 'prompts'). Trained on text-image pairs culled from the Internet, DALL-E claims to create 'realistic imagery and art' in multiple styles and compositions.

Developed by OpenAI and first revealed in January 2021, DALL-E uses a modified version of large language model GTP-3 to generate images. DALL-E2, which generates more diverse, higher resolution images faster, was released in May 2022. 

In September 2023, OpenAI released DALLE-3 with improved capabilities, added guardrails, and integration with ChatGPT.

DALL-E has been praised by researchers and commentators for the ease with which it makes it possible to create highly realistic, if surprising and weird, images and artwork at high speed. 

Risks and harms 🛑

DALL-E has been criticised for creating inappropriate and harmful content, including misinformation and disinformation, amplifying biases, violating intellectual property rights, destroyiand damaging the environment, amongst others.

Transparency and openness 🙈

OpenAI's refusal to let third parties assess its algorithm makes it difficult to understand how it works, and how its risks can be managed.

• Given the variety and nature of the risks of DALL-E, and its potential negative impacts, OpenAI's decision to restrict user access to DALL-E has mostly been welcomed, even if some users complain that Stable Diffusion, Midjourney and other image generation tools are open to everyone and can be used with few, if any, restrictions

• In July 2022, OpenAI announced DALL-E 2 would be made available to up to one million users as part of a large-scale beta test. An API for the system was released in November 2022

Incidents and issues 🔥

• Accuracy: DALLE's ability to produce illogical and incomprehensible content, especially when presented with longer prompts.

• Bias: DALL-E exhibits and reinforces biases, including gender, racial and cultural stereotyping.

• IP/copyright: DALL-E produces content that copies existing artwork, and is reckoned to abuse copyright and trademarks. In November 2023, Disney cracked down on Microsoft's DALL-E-powered AI image creator system after users created posters featuring their dogs as the stars of Pixar Studio films, violating the media company's intellectual property.

• Employment: DALL-E's will kill the careers of artists, graphic designers, animators, anime cartoonists, food photographers and others.

• Environment: Generative models like DALL-E typically consume huge amounts of energy.

• Mis/disinformation: Having stopped users uploading and editing human facial images in order to minimise the generation of deepfakes, OpenAI'S decision to reintroduce this ability has fueled concerns that it is much easier to use DALL-E to generate and spread mis and disinformation. UC Berkeley researcher Henry Farid reckons DALL-E 'could be disinformation on steroids'.

• Privacy: OpenAI is trained on photographs and other images images publicly available on the internet without consent. Furthermore, the company's decision to reintroduce the ability to upload third-party faces is seen to potentially damage the privacy of people whose consent may not have been obtained.

• Safety: DALL-E can produce offensive or explicit content, as well as content that can be construed as harrassment or bullying.

These limitations and risks broadly reflect those published by OpenAI upon DALL-E's launch and DALLE-2 upgrade.

Research, advocacy 🧮

• Cho J., Zala A., Bansal M. (2023). DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Models

• Luccioni A.S., Akiki C., Mitchell M., Jernite Y. (2023). Stable Bias: Analyzing Societal Representations in Diffusion Models

• Leivada E., Murphy E. Marcus G. (2022). DALL-E 2 Fails to Reliably Capture Common Syntactic Processes

• Federico Bianchi F. et al (2022). Easily Accessible Text-to-Image Generation Amplifies Demographic Stereotypes at Large Scale

News, commentary, analysis 🗞️

• https://www.vox.com/future-perfect/23023538/ai-dalle-2-openai-bias-gpt-3-incentives

• https://www.technologyreview.com/2021/01/05/1015754/avocado-armchair-future-ai-openai-deep-learning-nlp-gpt3-computer-vision-common-sense/

• https://techcrunch.com/2022/07/22/commercial-image-generating-ai-raises-all-sorts-of-thorny-legal-issues/

• https://www.npr.org/2022/07/20/1112331013/dall-e-ai-art-beta-test

• https://www.bloomberg.com/opinion/articles/2022-04-21/openai-project-risks-bias-without-more-scrutiny

• https://www.dezeen.com/2022/04/21/openai-dall-e-2-unseen-images-basic-text-technology/

• https://www.vox.com/recode/23405149/ai-art-dall-e-colonialism-artificial-intelligence

• https://www.engadget.com/dall-e-generative-ai-tracking-data-privacy-160034656.html

• https://www.washingtonpost.com/business/openai-project-risks-bias-without-more-scrutiny/2022/04/21/4876513a-c13d-11ec-b5df-1fba61a66c75_story.html

• https://www.vice.com/en/article/g5vbx9/dall-e-is-now-generating-realistic-faces-of-fake-people

• https://www.wsj.com/articles/think-of-any-image-then-ask-an-ai-art-generator-for-it-the-results-are-amazingand-terrifying-11666179308

• https://www.theverge.com/2022/9/21/23364696/getty-images-ai-ban-generated-artwork-illustration-copyright

• https://venturebeat.com/business/openai-will-dall-e-2-kill-creative-careers/

• https://newatlas.com/computers/dall-e-2-ai-art/

• https://www.cosmopolitan.com/lifestyle/a40314356/dall-e-2-artificial-intelligence-cover/

• https://www.nytimes.com/2022/04/06/technology/openai-images-dall-e.html

• https://www.nytimes.com/2022/08/24/technology/ai-technology-progress.html

• https://www.getrevue.co/profile/themarkup/issues/confronting-the-biases-embedded-in-artificial-intelligence-1137376

• https://www.businessinsider.com/ai-image-generators-artists-copying-style-thousands-images-2022-10

• https://eandt.theiet.org/content/articles/2022/11/full-power-behind-ai-s-green-ambitions/

• https://spectrum.ieee.org/openai-dall-e-2

• https://adityaramesh.com/posts/dalle2/dalle2.html

• https://www.theregister.com/2022/05/08/in_brief_ai/

• https://www.theguardian.com/technology/2022/may/04/techscape-openai-dall-e-2

• https://edition.cnn.com/2022/10/21/tech/artists-ai-images/

• https://www.theverge.com/2022/9/28/23376328/ai-art-image-generator-dall-e-access-waitlist-scrapped

• https://gizmodo.com/dall-e-ai-openai-deep-fakes-image-generators-1849557604

• https://restofworld.org/2022/ai-backlash-anime-artists/