What happened

Researchers from Answer.AI spent a month testing Devin and reported that out of 20 attempted tasks, 14 failed, 3 were inconclusive, and only 3 succeeded.

The AI tool struggled to create new projects, perform research tasks and analyse existing code, and often produced overly complex, unusable solutions and spent days pursuing impossible tasks. It also failed to interpret user commands correctly and continued running, producing errors in the output.

Specifically, when asked to deploy multiple applications to deployment platform Railway, Devin failed to recognise this wasn't possible and spent over a day attempting nonviable approaches. The tool also became trapped in endless cycles of trying to parse HTML when given web scraping tasks. 

Furthermore, security reviews resulted in numerous false positives and hallucinated issues.

Why it happened

The huge gap between Devin's promised and actual capabilities indicates inadequate testing before release. 

It also may reflect pressure from investors - which include Founders Fund and Khosla Ventures - for quick returns from Devin developer and owner Cognition AI.