Replika

Transparency and accountability 🙈

Replika AI is seen to exhibit several transparency and accountability limitations:

• Algorithmic black box. Making it difficult for users to understand how decisions are made or responses are generated. 

• Data use. Replika claims not to use conversation content for marketing, but they mention using it for other "legitimate interests" like analysing service effectiveness and developing business strategies. The extent to which sensitive personal chat information is used to train AI models is not clearly outlined in the company's privacy policy.

• Data protection. There are concerns about Replika's legal basis for processing users' data, especially for minors who cannot legally consent.

• Impacts. There is a lack of clear information on how Replika addresses potential biases in its AI system. The app's impact on vulnerable users, particularly minors and those with emotional vulnerabilities, is not adequately addressed.

• Third-party audits. There appears to be no provision for third-party audits or independent validation of the AI system's performance and ethical standards.

Risks and harms 🛑

Replika has been associated with posing significant risks and harms to its users and others, including: 

• Privacy and data security. Including excessive data collection, the opaque use of user data for advertising, and weak security

• Emotional and psychological. Including anthropomorphism, addiction, emotional manipulation, and the exploitation of vulnerable people.

• Childrens' safety. Stemming from inadequate age verification and poor data processing.

• Social impact. AI companions replacing human social interactions poses behavioural and other broader societal damage.

Legal, regulatory 👩🏼‍⚖️

• Crown Prosecution Service (2023). Windsor Castle intruder pleads guilty to threatening to kill Her late Majesty Queen Elizabeth II

• Il Garante per la protezione dei dati personali (Feb 2023). Provvedimento del 2 febbraio 2023

• Il Garante per la protezione dei dati personali (Feb 2023). Artificial intelligence: italian SA clamps down on ‘Replika’ chatbot

Research, advocacy 🧮

• Mozilla Foundation. Privacy Not Included

• Ciriello R.F. et al. Ethical Tensions in Human-AI Companionship: A Dialectical Inquiry into Replika

• Maples B. et al. Loneliness and suicide mitigation for students using GPT3-enabled chatbots

News, commentary, analysis 🗞️

• https://futurism.com/chatbot-abuse

• https://thenextweb.com/news/confused-replika-ai-users-are-standing-up-for-bots-trying-bang-the-algorithm

• https://www.thesun.co.uk/tech/17368588/men-ai-girlfriends-abuse-online/

• https://fortune.com/2022/01/19/chatbots-ai-girlfriends-verbal-abuse-reddit/

• https://www.vice.com/en/article/z34d43/my-ai-is-sexually-harassing-me-replika-chatbot-nudes

• https://www.vice.com/en/article/n7zaam/replika-ceo-ai-erotic-roleplay-chatgpt3-rep

• https://futurism.com/the-byte/replika-chatbot-harassing-users

• https://www.thegamer.com/replika-role-play-romance-updates-controversy/

• https://www.reddit.com/r/replika/comments/iq3cuk/replika_and_sexual_consent/