BookCorpus large language dataset

BookCorpus (also known as Toronto Book Corpus) is a dataset that draws on 11,000+ free, unauthored books representing 16 different genres culled from Smashwords.com, a site that describes itself as 'the world’s largest distributor of indie ebooks'.

Compiled in 2014 by a group of University of Toronto and MIT researchers and funded by Google and Samsung, BookCorpus has been used to train influential large language models such as Google's BERT, Amazon's Bort, and OpenAI's GPT.

Risks and harms 🛑

BookCorpus has been accused of illegally and unethically violating copyright, undermining authors' ability to control distribution of their creative works, which is essential for their livelihood, exhibiting racial and religious bias, misleading marketing and poor transparency.

Transparency and accountability 🙈

Here are some key transparency limitations of the BookCorpus large language dataset:

• Source selection. Limited information is available about how books were selected for inclusion in the corpus, potentially introducing selection bias.

• Copyright and permissions. It is unclear whether proper permissions were obtained for all books included, raising legal and ethical concerns.

• Data cleaning process. Details about any preprocessing or cleaning steps applied to the text are not fully disclosed.

• Demographic representation. There is limited transparency regarding the diversity of authors and perspectives represented in the corpus.

• Language variety. While primarily English, the extent of dialect variation or inclusion of non-English text is not clearly specified.

• Versioning and updates. Clear information about different versions of the dataset and any updates made over time are limited.

• Quality control. Details about measures taken to ensure the quality and accuracy of the digitized text are not fully transparent.

• Metadata availability. The extent and accuracy of metadata (e.g., author information, publication details) accompanying the text is unclear.

• Duplicate content. Information about how potential duplicates or near-duplicates are handled within the corpus are lacking.

• Ethical review. There is no information regarding any ethical review process conducted during the dataset's creation.

Research, advocacy 🧮

• Bandy J., (2021). BookCorpus datacard/datasheet

• Bandy J., Vincent N. (2021). Addressing "Documentation Debt" in Machine Learning Research: A Retrospective Datasheet for BookCorpus
  
  

Investigations, assessments, audits 🧐

• Bandy J., Towards Data Science (2021). Dirty Secrets of BookCorpus, a Key Dataset in Machine Learning

News, commentary, analysis 🗞️

• https://towardsdatascience.com/dirty-secrets-of-bookcorpus-a-key-dataset-in-machine-learning-6ee2927e8650

• https://www.theguardian.com/books/2016/sep/28/google-swallows-11000-novels-to-improve-ais-conversation

• https://jack-clark.net/2021/05/17/import-ai-249-ibms-massive-code-dataset-dataset-archaeology-bookcorpus-facebook-wants-computers-to-read-the-world/

• https://info.deeplearning.ai/the-batch-face-recognition-for-the-masses-labeling-libel-documenting-datasets-what-machines-want-to-see-1