Microsoft Copilot

Transparency, accountability 🙈

Microsoft Copilot is associated with a number of significant transparency and accountability limitations, thereby making it difficult for users, researchers, regulators and others to evaluate risks, assign responsibility, enforce standards, and build appropriate trust in the system. The limitations include: 

• Model opacity. Proprietary algorithms and training methods are not disclosed to users; Lack of clarity about which specific models power different Copilot features; o visibility into how the system makes decisions or generates responses; Users cannot audit the reasoning process behind outputs; Black box nature makes it difficult to understand why certain content is produced

• Training data. Limited disclosure about what data was used to train the models; Unclear sourcing and licensing of training materials; No transparency about how copyrighted, personal, or sensitive data was handled; Difficulty determining if specific content influenced the model's training; Lack of visibility into data filtering or curation processes.

• Decision-making. Unclear accountability when Copilot generates harmful, incorrect, or biased content; Difficulty determining whether errors stem from the model, training data, or user input; No clear chain of responsibility between Microsoft, enterprise customers, and end users; Ambiguous liability when AI-generated content causes harm or damages.

• Documentation. Limited information about system limitations and failure modes; Insufficient guidance on appropriate vs inappropriate use cases;  Vague explanations of how enterprise data is processed and protected; Unclear versioning and change logs when models are updated; Inadequate disclosure of when outputs may be unreliable.

• Audit and oversight. No standardised mechanisms for independent audits of the system; Difficult for organisations to verify compliance with their own policies; Limited tools for tracking and reviewing AI-generated decisions over time; Lack of explainability features for high-stakes applications;  Insufficient logging of AI interactions for accountability purposes.

• Governance structures. Opaque internal processes for content moderation and safety decisions; Limited external oversight or input into policy decisions; Unclear escalation paths when users identify problems; Insufficient transparency about how user feedback influences system changes.

• Performance. Lack of standardised benchmarks for evaluating real-world performance. Limited disclosure of accuracy rates, failure rates, or error patterns. No clear reporting on demographic performance disparities. Insufficient transparency about system changes and their impacts.

• Data usage and rights. Unclear terms about how user prompts and outputs are retained or used; Ambiguous policies on whether interactions train future models; Limited transparency about data sharing with third parties; Confusion about intellectual property rights for AI-generated content.

• Complaints and appeals. No clear process for contesting AI-generated outputs or decisions; Limited avenues for users to seek remedies for AI-caused harms; Unclear dispute resolution procedures; Difficulty obtaining explanations for specific outputs after the fact.

• Regulatory compliance. Limited transparency about compliance with emerging AI regulations;  Unclear how the system meets requirements in different jurisdictions; Insufficient disclosure for meeting industry-specific standards; Vague assurances about GDPR, CCPA, and other privacy law compliance.

Risks, harms 🛑

Microsoft's Copilot chatbot has been criticised for posing multiple risks and causing significant actual harms, including: 

• Accuracy and reliability. The generation of incorrect, outdated, or misleading information (hallucinations); potential for creating plausible-sounding but factually wrong code, documents, or analyses; users may over-rely on outputs without proper verification; Inconsistent performance across different tasks and domains.

• Bias and fairness. Perpetuation of biases present in training data; Potential for discriminatory outputs in hiring, performance reviews, or decision-making; Unequal performance across different languages and cultural contexts.

• Content generation. Can rapidly create convincing but false narratives, making fabricated content appear authoritative; Ability to generate misleading articles, reports, or social media posts at scale; Can produce synthetic "evidence" like fake meeting summaries, email chains, or document histories; Makes sophisticated disinformation campaigns more accessible to non-experts.

• Amplification of false information. May inadvertently spread misinformation by confidently presenting hallucinated facts; Can synthesise and legitimise conspiracy theories or fringe viewpoints when prompted; Difficulty for average users to distinguish between AI-generated falsehoods and accurate information; Creates a "laundering" effect where false claims gain credibility through professional-looking output.

• Manipulation and deception. Can be used to create deepfake text content that mimics specific writing styles or voices; Enables personalized disinformation targeting specific audiences or individuals; Facilitates astroturfing by generating fake grassroots content or reviews; Can produce misleading summaries that distort the meaning of source materials.

• Erosion of information trust. Contributes to general uncertainty about content authenticity ("Did a human or AI write this?"); Makes fact-checking more difficult as volume of AI-generated content increases; Potential to overwhelm information ecosystems with synthetic content; Reduces trust in legitimate sources when errors occur.

• Political and social. Can generate election misinformation, false voter information, or misleading campaign materials; Ability to create fake news articles or propaganda at unprecedented scale; Risk of manipulating public opinion on important issues; Potential interference in democratic processes.

• Security vulnerabilities. Risk of prompt injection attacks that manipulate the AI's behaviour; Potential generation of insecure code with vulnerabilities; Social engineering risks through AI-generated phishing content; Exposure of system prompts or internal logic through adversarial queries.

• Privacy and data protection. The collection and processing of sensitive user data including documents, emails, and chat histories; the potential exposure of confidential business information through prompts or generated outputs; unclear data retention policies and how information is used for model training; risk of data leakage between different organisational tenants.

• Workplace and economic impacts. Job displacement concerns for certain roles, particularly administrative and entry-level positions; Potential deskilling as workers become dependent on AI assistance; Pressure on workers to increase productivity beyond sustainable levels; Unequal access creating advantages for some employees over others.

• Compliance and legal issues. Copyright concerns regarding training data and generated content; Challenges meeting regulatory requirements in sensitive industries; Unclear liability for AI-generated errors or harmful outputs; Potential violations of professional standards in fields like law or medicine.

Research, advocacy 🧮

• ChatGPT/LLM Errors Tracker

• Greshake K., Abdelnabi S., Mishra S., Endres C., Holz T., Fritz M. (2023). More than you've asked for: A Comprehensive Analysis of Novel Prompt Injection Threats to Application-Integrated Large Language Models

• Gao C.A., Howard F.M., Markov N.S., Dyer E.C., Ramesh S., Luo Y., Pearson A.T, (2022). Comparing scientific abstracts generated by ChatGPT to original abstracts using an artificial intelligence output detector, plagiarism detector, and blinded human reviewers