GPT-2 large language model

GPT-2 is a large-scale, general purpose language model that generates text. Developed by OpenAI, GPT-2 was released as a limited beta in February 2019. The complete version, comprising 1.5 billion parameters, was published in November 2019.

Created to perform a variety of purposes, such as providing prompts, summarising text, and answering questions, GPT-2 was trained on BookCorpus and a dataset of 8 million web pages scraped from the internet.

Risks and harms 🛑

GPT-2 was seen to pose risks and harms by generating convincing fake news and biased content, and damaging to the environment. 

On its release, researchers, practitioners and commentators generally praised GPT-2's ability to generate realistic, plausible writing and translate text. Its flexibility was also welcomed.

Some were less enthusiastic, pointing out the model's tendency to generate gibberish, incoherent lengthy text, and to answer questions inaccurately. 

Others raised were potential misuses of GPT-2, including to generate convincing misinformation and disinformation, its ability to 'corrupt' people even when they are aware the source of advice is AI, and the huge amounts energy required to train and run GPT-2 and similar models.

Research, advocacy 🧮

• Leib M., Köbis N.C., Rilke R.M., Hagens M., Irlenbusch B. (2021). The corruptive force of AI-generated advice

• Kobis A., Mossink L.D. (2021). Artificial intelligence versus Maya Angelou: Experimental evidence that people cannot differentiate AI-generated from human-written poetry

• Strubell E., Ganesh A., McCullum A. (2019). Energy and Policy Considerations for Deep Learning in NLP

News, commentary, analysis 🗞️

• https://www.theverge.com/2019/11/7/20953040/openai-text-generation-ai-gpt-2-full-model-release-1-5b-parameters

• https://www.theverge.com/2019/2/14/18224704/ai-machine-learning-language-models-read-write-openai-gpt2

• https://www.theverge.com/2019/2/21/18234500/ai-ethics-debate-researchers-harmful-programs-openai

• https://www.vox.com/future-perfect/2019/2/14/18222270/artificial-intelligence-open-ai-natural-language-processing

• https://www.theguardian.com/technology/2019/feb/14/elon-musk-backed-ai-writes-convincing-news-fiction

• https://towardsdatascience.com/openais-gpt-2-the-model-the-hype-and-the-controversy-1109f4bfd5e8

• https://venturebeat.com/2019/02/22/ai-weekly-experts-say-openais-controversial-model-is-a-potential-threat-to-society-and-science/

• https://thegradient.pub/openai-please-open-source-your-language-model/

• https://thenextweb.com/artificial-intelligence/2019/02/26/whos-afraid-of-openais-big-bad-text-generator/

• https://www.wired.com/story/ai-text-generator-too-dangerous-to-make-public/

• https://www.technologyreview.com/2019/02/14/137426/an-ai-tool-auto-generates-fake-news-bogus-tweets-and-plenty-of-gibberish/

• https://www.newscientist.com/article/2194119-fears-of-openais-super-trolling-artificial-intelligence-are-overblown/

• https://www.popularmechanics.com/technology/infrastructure/a27793543/artificial-intelligence-carbon-footprint/

• https://www.technologyreview.com/2019/06/06/239031/training-a-single-ai-model-can-emit-as-much-carbon-as-five-cars-in-their-lifetimes/