BookCorpus, a dataset used to train dozens of influential language models including Google’s BERT, OpenAI’s GPT and Amazon’s Bort, was accused of abusing copyright, bias and misleading marketing. 

In a working paper, researchers Jack Bandy and Nicholas Vincent set out several concerns abut BookCorpus, and called for stronger standards for the documentation of datasets. 

Notably, they found that BookCorpus violated the copyright restrictions of over 200 books which had been taken from Smashwords, a website that describing itself as “the world’s largest distributor of indie ebooks.

The reseachers also discovered that the size of the BookCorpus dataset had been inaccurately described, with several books reproduced multiple times, and that the sample was skewed to certain genres and religious forms.

Despite these concerns, BookCorpus remains widely available on data sharing websites such as Hugging Face.