As we're still in the early days of building applications with foundation models, it's normal to make mistakes. But some pitfalls are so common that if you've worked on any AI product, you've probably seen them before. The good news is that they are avoidable.
Pitfall 1: Every Problem is a Nail
Every time there's a new technology, I can hear the collective sigh of senior engineers: "Not everything is a nail." Generative AI isn't an exception. Its seemingly limitless capabilities only exacerbate the tendency to use it for everything.
A team pitched me the idea of using generative AI to optimize energy consumption. They fed a household's list of energy-intensive activities and hourly electricity prices into an LLM to create a schedule. Their experiments showed a 30% reduction. I asked: "How does it compare to simply scheduling the most energy-intensive activities when electricity is cheapest?" They never followed up. I suspect that greedy scheduling is quite effective. Even if it's not, there are other cheaper and more reliable optimization solutions, like linear programming.
It can often be beneficial to explore a new approach to get a sense of what's possible, as long as you're aware that your goal isn't to solve a problem but to test a solution. "We solve the problem" and "We use generative AI" are two very different headlines.
Pitfall 2: AI is the Easy Part, Product is the Hard Part
Many people have told me that the technical aspects of their AI applications are straightforward. The hard part is user experience (UX). While we know that generative AI is changing how we read, write, learn, work, and entertain, we don't quite know how yet.
A few examples: a meeting transcript summarization app found that users didn't care about the summary, they only wanted action items specific to them. A skill fit assessment chatbot discovered that users didn't want correct responses, they wanted helpful responses. Users want tips on what the gaps are and how to close them. Intuit built a tax chatbot and found users hated typing, so they added suggested questions to click on.
Because everyone uses the same models nowadays, the AI components of AI products are similar, and the differentiation is product.
Pitfall 3: Shiny New Tools Distraction
Given so many shiny new technologies, it's tempting to jump straight into using them. Incorporating external tools too early can cause problems. It can lock you into a particular framework or API before you've proven the concept, and it can introduce bugs and unpredictable changes from tool developers.
Eventually, abstractions are good. But abstractions need to incorporate best practices and be tested over time. As we're still in the early days of AI engineering, best practices are still evolving, and we should be more vigilant when adopting any abstraction.
Pitfall 4: Underestimating the Journey from Demo to Product
A startup founder told me it took one month to achieve 80% of the experience they wanted, and an additional four months to surpass 95%. The initial success made them grossly underestimate how challenging it is to improve the product, especially around hallucinations. The challenges they faced were many: hard for tonal requests in the system prompt to be perfectly obeyed, and hard to create a specific set of unit tests because the combination of queries is basically infinite.
This is perhaps one of the first painful lessons anyone who has built an AI product quickly learns. It's easy to build a demo, but hard to build a product. Other issues include unreliability from API providers and legal risks around AI output copyrights, data access, and user privacy.
Pitfall 5: Forgoing Human Evaluation
To automatically evaluate AI applications, many teams opt for the AI-as-a-judge approach. A common pitfall is forgoing human evaluation to rely entirely on AI judges. While AI judges can be useful, they aren't deterministic. The quality of a judge depends on the underlying model, the judge's prompt, and the use case.
The teams with the best products all have human evaluation to supplement their automated evaluation. Human evaluation helps correlate human judgments with AI judgments, gain a better understanding of how users use your application, and detect patterns and changes in users' behaviors. Manual inspection of data has probably the highest value-to-prestige ratio of any activity in machine learning.
Pitfall 6: Crowdsourcing Strategy
This is a mistake I saw in the early days when enterprises were in a frenzy to adopt generative AI. Unable to come up with a strategy, many tech executives crowdsourced ideas from the whole company. "We hire smart people. Let them tell us what to do." They then try to implement these ideas one by one.
And that's how we ended up with a million text-to-SQL models and a billion code plugins. While it's indeed a good idea to listen to smart people, individuals might be biased toward problems that affect their day-to-day work instead of those that bring the highest ROI. Without an overall strategy, it's easy to get sidetracked into a series of small, low-impact applications and come to the wrong conclusion that gen AI has no ROI.