GoEmotions

Transparency and accountability 🙈

The GoEmotions dataset has several transparency limitations.

• Data collection process. There is limited information on how specific Reddit comments were selected for inclusion in the dataset.

• Annotator demographics. The dataset lacks detailed information about the demographics and backgrounds of the annotators, which could influence emotion labeling.

• Annotation guidelines. While some information is provided, the full set of detailed guidelines given to annotators is not publicly available.

• Inter-annotator agreement. While some metrics are provided, there's limited insight into specific areas of disagreement among annotators.

• Data cleaning process. The exact methods used to clean and preprocess the Reddit comments are not fully detailed.

• Excluded data. There is limited information on what types of comments or content were excluded from the dataset and why.

• Consent and privacy. It is unclear whether or how consent was obtained from the original comment authors, or how privacy concerns were addressed.

• Potential biases: While some biases are acknowledged, there may be insufficient detail on potential biases in the data selection or annotation process.

• Version control. Information about how the dataset might be updated or versioned over time is limited.

Risks and harms 🛑

GoEmotions has been criticised for its high rate of mislabeled data and for its unethical violation of the privacy of Reddit users by exploiting their content without acknowledgement or consent. 

Research, advocacy 🧮

• SurgeAI: 30% of Google's Emotions Dataset is Mislabeled