GPT-3

Report incident 🔥 | Improve page 💁 | Access database 🔢

GPT-3 (Generative Pre-trained Transformer 3) is a large language model that uses deep learning to generate natural, human-like language from text prompts.

Developed by OpenAI and released as a beta in May 2020, GPT-3 has 175 billion parameters, ten times more than previous models. ChatGPT was built using a GPT-3.5 model.

GPT-3 won praise from technology professionals, commentators, scientists, and philosophers for the quality of text it produces, and its ability to synthesise massive amounts of content. MIT Technology Review described it as 'shockingly good' and able to generate 'amazing human-like text on demand'. It is, according to the National Law Review, 'a glimpse of the future'.

Large language model

A large language model (LLM) is a computational model capable of language generation or other natural language processing tasks.

Source: Wikipedia 🔗

System 🤖

GPT-3 🔗
GPT-3 Wikipedia profile

Documents 📃

Operator: OpenAI; Microsoft
Developer: OpenAI

Country: USA

Sector: Multiple

Purpose: Generate text

Technology: Large language model (LLM); NLP/text analysis; Neural network; Deep learning; Machine learning
Issue: Governance; Accuracy/reliability; Bias/discrimination - multiple; Dual/multi-use; Safety; Employment; Environment; Transparency

Risks and harms 🛑

GPT-3 has been criticised for being inaccurate, biased, offensive, violating copyright, poor transparency, and for actual and potential job losses, and damage to the environment.

NYU professor Gary Marcus dismissed GPT-3 as 'a fluent spouter of bullshit, but even with 175 billion parameters and 450 gigabytes of input data, it’s not a reliable interpreter of the world. '

OpenAI CEO Sam Altman admitted the tool has 'serious weaknesses and sometimes makes very silly mistakes.'

Accuracy/reliability

GPT-3 has been shown to make basic errors, lack common sense and empathy, and repeat or contradict itself in lengthy passages of text.

The GPT-3 driven Historical Figures app produced multiple inaccurate and contradictory results, including having Nazi propaganda minister Joseph Goebbels say he 'did not hate Jews'.
Having developed its own version of GPT-3, French healthcare technology company Nabla deduced the model 'had no understanding of time or any proper logic'.
Facebook AI head Yan LeCun said the text generator is 'not very good' as a Q&A or dialogue system.

Bias/discrimination

GPT-3 has been shown to emit sexist and racist language, including anti-Muslim and anti-Semitic screeds.

Stanford McMaster University researchers found the model consistently associates Muslims with violence.

Copyright

GPT-3 was built using Common Crawl data, meaning it incorporates web pages, posts, copyrighted articles and books from the BBC, The New York Times, and others, according to TechCrunch.

Mis/disinformation

GPT-3 is prone to 'hallucinate' and invent realistic-looking information based on its training data.

For example, a student was able to publish fake blog posts using GPT-3 that passed as human, whilst another user was able to posts comments on Reddit for a week unnoticed.
t can also mimic the style of the QAnon conspiracy theory and create its own narrative that fits within the conspiracy theory, according to Georgetown University Center for Security and Emerging Technology researchers.

A July 2023 University of Zurich research study found that GPT-3 produces more 'compelling disinformation' than humans.

Safety

GPT-3 has been shown to be capable of spouting offensive and hateful content.

A version of GPT-3 developed by Nabla recommended a patient commit suicide.
A GPT-3 update resulted in AI Dungeon players typing words that caused the game to generate inappropriate stories.

Environment

Large language models such as GPT-3 have been criticised for their impact on carbon emissions, including by Google researchers (who subsequently lost their jobs).

University of Copenhagen researchers estimated that training GPT-3 in a data center fully powered by fossil fuels would have had roughly the same carbon footprint as driving a car to the moon and back.

Research highlighted in Stanford HAI's 2023 AI Index Report indicates GPT-3 released 502 metric tonnes of carbon during its training, and required 1,287 MWh of power - far more than any other machine learning model analysed.
A study (pdf) by University of California researchers estimates that GPT-3 's training alone consumed around 700,000 litres of water.

Employment

In January 2023, Time journalist Billy Perrigo revealed that OpenAI used Kenyan workers being paid less than USD 2 an hour to de-toxify GPT-3 and Open AI's ChatGPT.

According to Perrigo, 'the work’s traumatic nature eventually led Sama to cancel all its work for OpenAI in February 2022, eight months earlier than planned.'

Business model

In September 2020, Microsoft licensed 'exclusive' use of GPT-3's underlying code, allowing it to embed, repurpose, and modify the model as it pleases.

The deal was seen as further confirmation of Open AI's move away from its non-profit status and declared mission to 'ensure that artificial general intelligence benefits all of humanity' to a more commercial model.

It also reinforced concerns over Microsoft's potential sway over Open AI, and the concentration of power amongst a few technology companies.

Transparency and accountability 🙈

Open AI laid the ground for GPT-3 by publishing a research paper Language Models are Few-Shot Learners which highlighted a number of the risks and harms associated with its model.

However, the company's decision to grant access to the model to a few select researchers rather than release GPT-3's algorithm to the general public proved controversial, with some seeing it as 'unscientific' and 'opaque', whilst others saw it as necessary given its scale and potential for serious and widespread harm.

Equally, a number of commentators figured Open AI's decision to operate a de facto black box likely reflected its inability to fully understand its model, and a commercial desire to protect its IP.

Research, advocacy 🧮

Spitale G., Biller-Adorno N., Germani F. (2023). AI model GPT-3 (dis)informs us better than humans
Stanford HAI (2023). 2023 AI Index Report - 2.8 Environment (pdf)
Li P., Yang J., Islam M.A., Ren S. (2023). Making AI Less 'Thirsty': Uncovering and Addressing the Secret Water Footprint of AI Models (pdf)
Argyle L.P., Busby E.C., Fulda N., Gubler J., Rytting C., Wingate C. (2022). Out of One, Many: Using Language Models to Simulate Human Samples
Abid A., Farooqi M., Zou J. (2021). Persistent Anti-Muslim Bias in Large Language Models (pdf)
Bender E.M., Gebru T., McMillan-Major A., Mitchell M. (2021). On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?
Wolff Anthony L.F., Kanding B., Selvan R. (2020). Carbontracker: Tracking and Predicting the Carbon Footprint of Training Deep Learning Models (pdf)
Nabla (2020). Doctor GPT-3: Hype or Reality?

News, commentary, analysis 🗞️

Related 🌐

Page info
Type: System
Published: January 2023
Last updated: October 2024