GPT-3 large language model

GPT-3 (Generative Pre-trained Transformer 3) is a large language model that uses deep learning to generate natural, human-like language from text prompts. 

Developed by OpenAI and released as a beta in May 2020, GPT-3 has 175 billion parameters, ten times more than previous models. ChatGPT was built using a GPT-3.5 model.

GPT-3 won praise from technology professionals, commentators, scientists, and philosophers for the quality of text it produces, and its ability to synthesise massive amounts of content. MIT Technology Review described it as 'shockingly good' and able to generate 'amazing human-like text on demand'. It is, according to the National Law Review, 'a glimpse of the future'. 

But New York Times technology columnist Farhad Manjoo described GPT-3 as 'at once amazing, spooky, humbling, and more than a little terrifying,' and NYU professor Gary Marcus dismissed it as 'a fluent spouter of bullshit, but even with 175 billion parameters and 450 gigabytes of input data, it’s not a reliable interpreter of the world. ' 

OpenAI CEO Sam Altman admitted the tool has 'serious weaknesses and sometimes makes very silly mistakes.'

Risks and harms

Accuracy/reliability

GPT-3 is seen to be useful for summarising or simplifying text. Yet it has been shown to make basic errors, lack common sense and empathy, and repeat or contradict itself in lengthy passages of text.

• The GPT-3 driven Historical Figures app produced multiple inaccurate and contradictory results, including having Nazi propaganda minister Joseph Goebbels say he 'did not hate Jews'. 

• Having developed its own version of GPT-3, French healthcare technology company Nabla deduced the model 'had no understanding of time or any proper logic'. 

• Facebook AI head Yan LeCun said the text generator is 'not very good' as a Q&A or dialogue system.

Bias/discrimination

GPT-3 has been shown to emit sexist and racist language, including anti-Muslim and anti-Semitic screeds. 

• Stanford McMaster University researchers found the model consistently associates Muslims with violence.

Copyright

GPT-3 was built using Common Crawl data, meaning it incorporates web pages, posts, copyrighted articles and books from the BBC, The New York Times, and others, according to TechCrunch.

Mis/disinformation

GPT-3 is prone to 'hallucinate' and invent realistic-looking information based on its training data. 

• For example, a student was able to publish fake blog posts using GPT-3 that passed as human, whilst another user was able to  posts comments on Reddit for a week unnoticed. I

• t can also mimic the style of the QAnon conspiracy theory and create its own narrative that fits within the conspiracy theory, according to Georgetown University Center for Security and Emerging Technology researchers. 

• A July 2023 University of Zurich research study found that GPT-3 produces more 'compelling disinformation' than humans. 

Safety

GPT-3 has been shown to be capable of spouting offensive and hateful content. 

• A version of GPT-3 developed by Nabla recommended a patient commit suicide. 

• A GPT-3 update resulted in AI Dungeon players typing words that caused the game to generate inappropriate stories.

Environment

Large language models such as GPT-3 have been criticised for their impact on carbon emissions, including by Google researchers (who subsequently lost their jobs).

• University of Copenhagen researchers estimated that training GPT-3  in a data center fully powered by fossil fuels would have had roughly the same carbon footprint as driving a car to the moon and back. 

• Research highlighted in Stanford HAI's 2023 AI Index Report indicates GPT-3 released 502 metric tonnes of carbon during its training, and required 1,287 MWh of power - far more than any other machine learning model analysed.

•  A study (pdf) by University of California researchers estimates that GPT-3 's training alone consumed around 700,000 litres of water.

Business model

In September 2020, Microsoft licensed 'exclusive' use of GPT-3's underlying code, allowing it to embed, repurpose, and modify the model as it pleases.

The deal was seen as further confirmation of Open AI's move away from its non-profit status and declared mission to 'ensure that artificial general intelligence benefits all of humanity' to a more commercial model. 

It also reinforced concerns over Microsoft's potential sway over Open AI, and the concentration of power amongst a few technology companies.

Employment

In January 2023, Time journalist Billy Perrigo revealed that OpenAI used Kenyan workers being paid less than USD 2 an hour to de-toxify GPT-3 and Open AI's ChatGPT. 

According to Perrigo, 'the work’s traumatic nature eventually led Sama to cancel all its work for OpenAI in February 2022, eight months earlier than planned.'

Transparency

Open AI laid the ground for GPT-3 by publishing a research paper Language Models are Few-Shot Learners which highlighted a number of the risks and harms associated with its model. 

However, the company's decision to grant access to the model to a few select researchers rather than release GPT-3's algorithm to the general public proved controversial, with some seeing it as 'unscientific' and 'opaque', whilst others saw it as necessary given its scale and potential for serious and widespread harm.

Equally, a number of commentators figured Open AI's decision to operate a de facto black box likely reflected its inability to fully understand its model, and a commercial desire to protect its IP.

Research, advocacy

• Spitale G., Biller-Adorno N., Germani F. (2023). AI model GPT-3 (dis)informs us better than humans

• Stanford HAI (2023). 2023 AI Index Report - 2.8 Environment (pdf)

• Li P., Yang J., Islam M.A., Ren S. (2023). Making AI Less 'Thirsty': Uncovering and Addressing the Secret Water Footprint of AI Models (pdf)

• Argyle L.P., Busby E.C., Fulda N., Gubler J., Rytting C., Wingate C. (2022). Out of One, Many: Using Language Models to Simulate Human Samples

• Abid A., Farooqi M., Zou J. (2021). Persistent Anti-Muslim Bias in Large Language Models (pdf)

• Bender E.M., Gebru T., McMillan-Major A., Mitchell M. (2021). On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?

• Wolff Anthony L.F., Kanding B., Selvan R. (2020). Carbontracker: Tracking and Predicting the Carbon Footprint of Training Deep Learning Models (pdf)

• Nabla (2020). Doctor GPT-3: Hype or Reality?