Occurred: August 2023
ChatGPT wrongly answers over half of software engineering questions it receives, according to a research study.
Purdue University researchers analysed how ChatGPT responded to 517 questions posed on Stack Overflow to assess the correctness, consistency, comprehensiveness, and conciseness of the chatbot's answers using linguistic and sentiment analysis, and by questioning a dozen volunteer participants.
The analysis showed that 52 percent of ChatGPT answers are incorrect, and 77 percent are verbose. 'Nonetheless', the researchers said, 'ChatGPT answers are still preferred 39.34 percent of the time due to their comprehensiveness and well-articulated language style.'
The paper raised questions about ChatGPT's ability to generate high quality engineering information.
Operator: Stack Overflow; OpenAI
Purpose: Generate text
Technology: Chatbot; NLP/text analysis; Neural network; Deep learning; Machine learning
News, commentary, analysis
Published: November 2023