Academia.eduAcademia.edu

Outline

Why AI Projects Fail: Lessons From New Product Development

https://doi.org/10.1109/EMR.2024.3419268

Abstract

AI projects have an alarming failure rate! A recent Deloitte investigation finds that only 18 to 36% of organizations achieve their expected benefits from AI (Mittal et al., 2024), and only 53% of AI projects proceed from prototype to production (Masci, 2022). Another report finds 87% never make it into production (Dilmegani, 2024). The "pilot paralysis" phenomenon, where companies undertake AI pilot projects but struggle to scale up, is epidemic (Gregory, 2021). Some estimates place the failure rate as high as 80%, almost double the failure rate of IT projects a decade ago (Bojinov, 2022), and higher than the failure rate for new product development (NPD) (Knudson et al., 2023). A Harvard Business Review article points out that almost all failure causes are "dumb reasons" (Tse et al., 2020), the result of poor business practices, and thus can be avoided. 1.1 Parallels Between AI Deployment and New Product Development Similarities exist between AI projects and B2B NPD projects: In both, the ultimate goal is to deliver a solution that meets the needs of the end-users or customers, whether they are internal (for AI projects) or external (for NPD projects). The activities in AI and NPD projects share many commonalities, such as crafting a business case, developing the solution, pilots or field trials, and scale-up or launch. (Here NPD refers to developing physical products, not software or services). "Those who cannot remember the past are condemned to repeat it" (Santayana, 1905) is an appropriate lesson for AI deployment. In NPD, failure causes have been well-researched. The first significant NPD study in 1964 sought managers' opinions on causes of failure; the top two were "inadequate market knowledge" and "technical defects in the product" (NICB, 1964). In 1975, an analysis of product failure cases identified a "lack of understanding of the marketplace (customers and competition) followed by "technical difficulties with the product" (Cooper, 1975). While personal opinions are many, the reasons for AI failure have yet to be revealed by robust research studies. Many of the challenges in NPD, such as understanding user needs or dealing with technical risks, are also common to AI projects, and the limited evidence available suggests AI failure reasons parallel the well-known reasons for NPD. 2. THE SEVEN REASONS FOR AI PROJECT FAILURES Armed with information from physical NPD together with evidence from the few but limited studies of AI projects (supplemented by opinion-based articles), the following important reasons for AI failure are delineated, shown in Fig. 1: 2.1 Failure to Understand Users' Needs, a Lack of Clear Objectives Most AI journeys begin with a technology-first orientation, what Overby calls the "shiny things disease" (Overby, 2020). Instead of starting with a solution looking for a problem, companies must start by investigating the specific user problem and then determine which AI tool solves it (Lamarre, 2023; Dilmegani, 2024). A McKinsey study highlights the importance of focusing on the users' needs and problems rather than being enamored by the technology itself (McKinsey, 2023). As in NPD, a lack of understanding of users' needs is a major reason for AI failure (Lamarre 2023). A lack of understanding of users' needs often underlies ill-defined project objectives. As in NPD, not conducting user research and failing to involve end-users throughout the development process leads to solutions that miss the mark, making this one of the reasons in Fig. 1 (McKinsey 2023; Lamarre, 2023). Without a deep understanding of the users' pain points, workflows, and applications, the resulting AI solution may fail to provide meaningful value or seamlessly integrate into existing processes. Recommendations: AI projects should begin with a comprehensive Voice of Process (VoP) and Voice of Business (VoB) study, similar to a Voice of Customer (VoC) study in NPD-gathering insights from various users to identify their needs, problems, and wants. And involving end-users throughout the development process by employing iterative feedback loops with demo validations, the project team (the AI Ops Team) ensures that the AI solution addresses problems, meets the users' expectations, and has value-in-use (Cooper, 2017, p. 317). Another best practice is to ensure the AI Ops Team is cross-functional with representation from the user group. The project's goals should then be defined, including the problem to be solved, the expected outcomes, and measurable success criteria.

FAQs

sparkles

AI

How did technical risks contribute to AI project failures?add

The IDC study indicates that 35% of firms cite model inaccuracy as a primary failure reason, surpassing cybersecurity concerns. Technical issues like data quality and model instability are significant contributors to these failures.

What explains the challenge of data quality in AI implementations?add

Roughly 80% of the effort in Machine Learning involves data preparation, and insufficient quality sources lead to inaccurate outputs. The lack of a production-ready data pipeline is ranked as the second most common reason for AI failure.

When did unrealistic expectations begin to impact AI project success rates?add

Research shows that 99% of AI project failures relate to overestimated expectations concerning technology capabilities. This problem has become prevalent as management seeks immediate solutions for complex problems without acknowledging the iterative AI development process.

Why do talent shortages significantly affect AI project outcomes?add

29% of firms report that a lack of skilled AI professionals leads to project failures. Companies face long delays and compromised quality due to the high costs associated with acquiring and managing talent in this field.

How can cross-functional teams improve AI project success?add

Interdisciplinary collaboration allows AI solutions to align with organizational goals and user needs. Research emphasizes that diverse teams help integrate AI seamlessly into existing business processes, counteracting siloed efforts that lead to inefficiency.

References (23)

  1. Agile Alliance. 2024. Agile Manifesto and Agile Mindset. Agile Alliance website. [Online]. Available: Download the Agile Manifesto as a PDF | Agile Alliance AIM. 2023. "Culture key to successful AI transformation: Empowering organizations for the future." AIM Research. June
  2. Online]. Available: Culture key to successful AI transformation: Empowering organizations for the future (aimresearch.co)
  3. Bratton, A. and Glynn, K. 2024. "Why your AI project is going to fail." Lextech. [Online]. Available: Why Your AI Project is Going to Fail (lextech.com)
  4. Brem, A. and Cooper, R.G. 2024. "Artificial Intelligence in new product development: An adoption and deployment process model for engineering management." In review: IEEE Transactions on Engineering Management. Article #7 [Online]. Available: Robert G. Cooper -Artificial Intelligence in NPD (bobcooper.ca
  5. Brown, S. 2021. "3 requirements for successful artificial intelligence programs." Jan 6. MIT Management. Oct 3 requirements for successful artificial intelligence programs | MIT Sloan
  6. Cooper, R.G. 1975. "Why new industrial products fail." Industrial Marketing Management 4: 315-26. [Online]. Available: ( Why new industrial products fail -ScienceDirect
  7. Cooper, R.G. 2017. Winning at New Products: Creating Value Through Innovation, 5 th edition, New York, NY: Basic Books, Perseus Books Group. [Online]. Available: Amazon.com : winning at new products Cooper. R.G. 2023. "New Products-What Separates the Winners from the Losers and What Drives Success." In: The PDMA Handbook of Innovation and New Product Development, 4th ed., edited by Bstieler, L. and Noble, C.H. . Chapter 1: Hoboken, NJ: Wiley. [Online]. Available: The PDMA Handbook of Innovation and New Product Development: Bstieler, Ludwig, Noble, Charles H.: 9781119890218: Amazon.com: Books
  8. Cooper, 2024. "Overcoming roadblocks to AI adoption in new product development." In process, Research-Technology Management. Article #9 [Online]. Available: Robert G. Cooper -Artificial Intelligence in NPD (bobcooper.ca)
  9. Creasey, T. 2023. "A point of view on AI, change, and change management." Change Success Insights. June 23. [Online].
  10. Available: (24) A Point of View on AI, Change, and Change Management | LinkedIn Dilmegani, C. 2024. "Why does AI fail?: 4 reasons for AI project failure in 2024." AI Multiple Research, Feb. 14. [Online]. Available: (https://research.aimultiple.com/ai-fail/ Globant. 2024. "The AI Manifesto." Globant blog. [Online]. Available: AI Manifesto | Globant Guinness, H. 2024. "What is Perplexity AI." Zapier Blog. April 3. [Online]. Available: What is Perplexity AI? How to use it + how it works (zapier.com)
  11. Haugen, K.S. 2021. "Failure in AI projects: What organizational conditions and how will managements' knowledge, organization and involvement contribute to AI project failure." Master's thesis, University of South-Eastern Norway, Faculty of School of Business, Spring. [Online]. Available: (https://openarchive.usn.no/usn- xmlui/bitstream/handle/11250/2783842/no.usn%3Awiseflow%3A2613763%3A44363626.pdf?sequence=1
  12. Jyoti, R., and Kuppuswamy, R. 2023. "Create more business value from your organizational data." IDC Research InfoBrief, Feb. [Online]. Available: (URL has spelling errors; ignore errors): Jyloti -imapctrs ofAI.pdf
  13. Knudsen, M.P., von Zedtwitz, M., Griffin, A., and Barczak, G. 2023. Best practices in new product development and innovation: Results from PDMA's 2021 global survey. Journal of Product Innovation Management 40(3), 257-275: Doi.org/10.1111/jpim.12663
  14. Korolov, M. 2023. "4 reasons why gen AI projects fail." CIO. March 4. [Online]. Available: (4 reasons why gen AI projects fail | CIO Haugen, E. 2023. "In Digital and AI transformations, start with the problem not the technology." McKinsey Strategy & Corporate Finance Practice, Nov. How to succeed in digital and AI transformations | McKinsey Masci, J. 2022. "AI has a poor track record, unless you clearly understand what you're going for." Industry Week, Jan. 19. AI Has a Poor Track Record, Unless You Clearly Understand What You're Going for | IndustryWeek McKinsey. 2023. "The state of AI in 2023: AI's breakout year." Quantum Black, August 1. [Online]. Available: The state of AI in 2023: Generative AI's breakout year | McKinsey NICB (National Industrial Conference Board). 1964. "Why new products fail," The Conference Board Record.
  15. Nylund, P.A, Ferràs-Hernández, X., and Brem, A. 2023. "A trust paradox may limit the application of Al-generated knowledge." Research-Technology Management (66)5: 44-52. Doi.org/10.1080/08956308.2023.2236475
  16. Nieto-Rodriguez, A., and Vargas, R. V. 2023. "How AI will transform project management." Harvard Business Review, Feb.
  17. Online]. Available: https://hbr.org/2023/02/how-ai-will-transform-project-management
  18. Overby, S. 2020. 8 reasons AI projects fail. The Enterprisers Project. March 4. [Online]. Available: https://enterprisersproject.com/article/2020/3/why-ai-projects-fail-8-reasons PDMA (Product Development and Management Association). 2023. The PDMA Handbook of Innovation and New Product Development, 4th ed., edited by Bstieler, L. and Noble, C.H. . Hoboken, NJ: Wiley. [Online]. Available: The PDMA Handbook of Innovation and New Product Development: Bstieler, Ludwig, Noble, Charles H.: 9781119890218: Amazon.com: Books
  19. Santayana, G. 1905. The Life of Reason, Volume 1, Chapter XII. https://archive.org/details/dli.ernet.864
  20. Schuerman, D. 2023. "It's time for every company to establish its own AI Manifesto." Forbes. Dec. [Online]. Available: It's Time For Every Company To Establish Its Own AI Manifesto (forbes.com)
  21. Schlegel, D., Schuler, K., and Westenberger, J. 2023. "Failure factors of AI projects: results from expert interviews." International Journal of Information Systems and Project Management (11)3: 25-40. Doi.org/10.12821/ijispm110302
  22. Tadiparthi,G. 2024. "Why AI projects fail." InsiteAI. [Online]. Available: Why AI Projects Fail -Insite AI Insite AI Tse, T., Esposito, M., Mizuno, T., and Goh, D. 2020. "The dumb reason your AI project will fail." Harvard Business Review, June 8. [Online]. Available: The Dumb Reason Your AI Project Will Fail (hbr.org)
  23. Westenberger, J., Schuler, K., and Schlegel, D. 2022. "Failure of AI projects: understanding the critical factors." Procedia Computer Science 196: 69-76 (https://creativecommons.org/licenses/by-nc-nd/4.0