The chatbot nobody asked for
Most failed AI features I have seen are chatbots nobody asked for.
The product teams I talk to tell the same story. Leadership watched a GPT demo and decided the product needed AI. Three months later there is a chatbot in the corner of the app. 3% of users have tried it and 0.5% use it regularly.
I have been on both sides of this. I have built AI features users loved, and I have killed features that looked brilliant in demos and died in production. This post is about the difference.
Why chatbots are the wrong default
Teams default to chatbots because they are visible. They look like AI, and stakeholders can see at a glance what changed. For most products they are still a poor default.
They make users change behavior. Your users already learned your UI. A chatbot asks them to stop clicking and start typing sentences. That adds friction.
Conversation is slow. Typing "show me last month's sales" takes longer than picking a date and clicking a report button. The product got worse.
Errors are awkward. When form validation fails, you show a red border. When a chatbot misunderstands, the user gets an odd answer, and each one costs a little trust.
The happy path is narrow. Chatbots demo well because demos follow a script. Real users ask things nobody planned for.
Chatbots are sometimes the right call. Customer support, complex search and use cases that are actually conversational can benefit from them. But "we should add AI" should not automatically mean "we should add a chatbot."
AI features that work
The best AI features are mostly invisible. The product gets smarter and the user does less.
Smart defaults and autocomplete
For most products this is the highest-return AI feature.
Without AI: the user fills out a 12-field form to create a report.
With AI: 8 fields are pre-filled from their past behavior. They adjust 2 or 3 and submit.
That saves them most of the work, and they never had to learn a new way of interacting. Gmail's Smart Compose is the standard example. It does not ask you to chat; it helps you type faster.
For one client, we added predictive field completion to their data entry flows. Form completion time dropped 40%. Nobody needed a tutorial. The feature showed up and helped.
Background processing and summaries
Do not wait for users to ask for a summary. Generate it ahead of time:
- When a document is uploaded, extract the key entities and show them.
- When a meeting recording finishes processing, show the highlights.
- When data changes significantly, send a short plain-language summary.
The rule is to do the AI work before the user asks. When they open the dashboard, the insight is already there.
Anomaly detection that points to an action
Most AI anomaly detection is useless because it reports noise. "Revenue increased 12%" is not news if it is normal seasonality.
Useful anomaly detection does three things:
- Filters out known patterns such as seasonality and promotions.
- Ranks by business impact.
- Suggests a specific action.
We rebuilt an alerting system around these rules. Instead of "metric X changed," users now see something like: "Conversion rate dropped 18% in the mobile checkout flow. This is unusual and is affecting daily revenue. The drop started after deploy #1234."
Someone can act on that. It is worth interrupting them for.
Generation with context
"Generate marketing copy" is a feature. "Pre-draft an email based on this customer's history and your earlier conversations" is a useful feature.
- Blank slate plus AI gives generic output.
- Context plus AI gives relevant output.
Once the model knows what the user is working on, what they did before and what they are trying to achieve, generation stops being a gimmick.
Build or buy
Every AI feature has three ways to get built.
1. API calls: fastest, least differentiated
Call OpenAI, Anthropic or another provider directly. This fits:
- Quick prototypes
- Features where the AI is not the main value
- Deadlines where you need to ship now
Cost: $0.01 to $0.10 per query, depending on model and tokens.
2. Fine-tuning: more effort, better quality
Train a model on your data for a specific task. This fits:
- Domain-specific language such as legal, medical or technical text
- Output that must follow a consistent format
- Cutting cost at scale
Cost: training plus inference, and inference is often cheaper than the base model.
3. Training your own model: almost never worth it
Unless you operate at Google or Anthropic scale, don't. Even if you have ML engineers, their time is better spent on fine-tuning and application logic than on training foundation models.
About 90% of the AI features I ship are API calls with good prompts. The other 10% are fine-tuned for classification tasks where we needed a consistent output format. I have not trained a model from scratch, and I probably never will.
User research breaks down for AI
Standard user research does not work well for AI features, because users do not know what they want.
Asking "would you use an AI feature that does X?" gets meaningless answers. Users picture the best case and say yes. In production they hit edge cases and stop using it.
Here is what works instead.
Watch behavior. Ship the feature as a quiet experiment and measure adoption over at least 4 weeks. The signal is who is still using it after the first week.
Measure task completion. The question that matters is "did you finish the task faster with suggestions on?"
A/B test with care. AI features often get a novelty bump. Test against a holdout group and measure over a longer period than you would for a UI change.
Listen for specific complaints. "This is cool" is noise. "It always gets company names wrong" is signal.
When to kill an AI feature
This is the hard part. You built something interesting, it took months of engineering, leadership is invested, and nobody uses it.
Signs it is time to kill it:
- Adoption flatlines after launch. A spike at release, then nothing. Users tried it and it did not stick.
- Engagement is shallow. Users trigger the feature but do not act on what it produces.
- Support tickets mention it negatively. "How do I turn off the AI suggestions?"
- It is slow and users work around it. They have learned to ignore it.
The sunk cost pull is real. I killed an AI feature after 4 months of development because adoption was 2% and falling. It hurt. But maintaining a feature nobody uses costs more than admitting it failed.
Treat it as something you learned. Write down what did not work, share it and don't repeat it.
Avoiding an AI winter inside your product
I have seen this sequence more than once:
- The company ships AI features with a lot of noise.
- The features underperform.
- Leadership loses interest.
- AI work gets deprioritized.
- Good AI ideas get lumped in with the failed ones.
That is an AI winter at company scale. Here is how I try to avoid it.
Start small and prove value. One well-built smart default beats ten chatbot experiments.
Set realistic expectations. AI features are less accurate than traditional software. If stakeholders expect 99.9% accuracy, they will be disappointed by 90%, even when 90% is excellent for the use case.
Measure the right thing. "Users tried the feature" is not success. "Users finished tasks faster" is.
Always leave a manual path. Every AI feature needs a fallback to doing it by hand. Users tolerate AI mistakes when they can correct them easily.
Where to look for AI work
The useful question is not "where can we add AI?" It is "where are users doing tedious thinking work that we could automate?"
Asking it that way changes what you build: smart defaults, and assistance grounded in the user's context. Those are the features that hold up after launch.
The hype cycle rewards announcements. Product work rewards value that lasts. The two often pull against each other, and I build for the second.