Stop Bolting a Chatbot Onto Your Product: A Practical Guide to Shipping AI Features

Most AI requests that land on an engineer's desk aren't about building a brand-new AI product. They're about a mature product that already has paying users and years of accumulated database schema, and leadership has been told to "add AI" before competitors do. The push sometimes comes from the board, sometimes from the sales team, sometimes from a customer's offhand remark on a renewal call.

The first version is almost always the same: a chat assistant in the bottom-right corner that lets users talk to the product. The demo is stunning. Then within a few weeks usage falls off a cliff — because users didn't open this product to chat, they opened it to get a job done.

As an engineer, what I find interesting is that the model is the easiest part of the whole thing. The real work is in the boring layers around it: which data goes in, whose permissions it runs under, where the output lands on the screen, and what happens when the model is wrong or goes down.

Start from the task, not the model

"Where can we use AI?" is the wrong question. The useful question is: what task do users do repeatedly in this product that is slow, tedious, or error-prone?

The candidates that consistently work share a common shape:

• Summarize something long: a thread, history, or recording a user must read through before acting.
• Fill structured fields from messy information: map an uploaded file, email, or note into a form you already have.
• Draft a first version of a reply or report, for a human to edit and approve.
• Classify and route: sort incoming items into the right queue or category.
• Find that thing: search that understands intent, not just keyword matching.

Each of these maps to an existing workflow you can measure today. That gives you a baseline, and the baseline is the only way you'll know whether the feature actually helped.

The author's own app, LectureNotes AI, is a clean case, even though it had AI from day one. Nobody records a lecture to chat with a model — they want notes they can study from. Transcription, key point extraction, and structured outlines all serve that one task, and none of them are a chat window.

Why you need an eval set. AI feature output is probabilistic. The same prompt drifts across models and versions, and vendors quietly update the models behind the scenes. Without a fixed eval set, a team can't tell whether a change is a real improvement or a regression, and can't catch it when model behavior shifts. The approach: pick 30 to 50 real inputs from your own product, write down in advance for each one what counts as a good output, and turn that into a repeatable benchmark. It doubles as a regression test, turning decisions like "switch to a cheaper model" into a config change plus an eval run, rather than a nerve-wracking rewrite.

The data work nobody puts in the estimate

Mature products are designed to show data to humans, not hand it to a model. That gap is where most of the development time goes.

Assembling context is ordinary engineering, and it's the bulk of the work. For a "summarize this customer's history" request, someone has to decide which records count as history, how far back to go, which fields matter, and which to exclude. In a product that's been running for a few years, that data is scattered across multiple tables, several services, and at least one third-party tool. Retrieval quality determines answer quality far more than model choice does.

AI features must respect your existing permission system. This is the one to underline. If a user can't open a record, the assistant can't summarize it, cite it, or let it leak into a search answer. The tempting shortcut is to give the AI feature a separate service account with broad read access, because that's faster to build. Don't. Fetch context as the requesting user, through the same permission layer the rest of the product uses.

The code example shows this "same permission layer, fail quietly" shape: the AI panel reuses the product's own access layer and never blocks the page it lives on. Two key points. First, there's no privileged path to the database, so permission bugs in the AI feature are structurally impossible rather than prevented by code review. Second, a slow or failing vendor returns null instead of freezing the screen. The product worked before this feature existed, and it should keep working when the feature is unavailable.

Be clear about where data is allowed to flow. If your customers have contracts with data processing terms, sending their records to a model vendor may or may not be covered. Figure that out before you build. If it isn't covered, there are workable options: zero-retention vendor terms, specific processing regions, self-hosted models, or stripping sensitive fields before the call.

What "run as the requesting user" means. Many AI features are designed with the model call sitting in a separate backend service that, for convenience, reaches the database with a service account holding broad read permissions. This looks fine in a demo, but it creates a privileged path that bypasses the product's permission system: any request that reaches that endpoint might read records the user shouldn't see. The right approach is to pass the identity and permission context of the current requesting user all the way down to the data access layer, so the AI feature reuses the same authentication and filtering logic the rest of the product already implements. That makes permission containment a structural guarantee rather than something that depends on catching gaps in code review each time.

Zero-retention vendor terms and regional processing. Beyond their default data retention policies, major model vendors usually offer enterprise-grade "zero retention" options promising not to store or train on customer request data, often at the cost of a higher unit price or an extra contracting process. Regional processing requires inference to happen in a specific jurisdiction (such as within the EU) to satisfy data localization compliance. Self-hosting open-source models is another path, keeping data entirely within your own infrastructure, but it comes with inference costs and operational burden. Which path to choose should be decided based on customer contract terms before work starts, not discovered after launch that it doesn't meet requirements.

Put it where the work already happens

Placement is the single biggest factor in whether anyone keeps using an AI feature. The patterns that work put the output in the screen the user is already on, at the moment they need it:

• A summary panel at the top of a record they just opened.
• Form fields pre-filled from an uploaded document, visually marked as suggestions before confirmation.
• A "draft reply" button next to the existing reply box, not a separate tool.
• A suggested category already selected in the dropdown they were about to click.

In every case, the existing workflow stays intact and the user stays in control. If the AI is wrong, they fix it the way they fix their own mistakes. If the AI is right, they save time and learn nothing new.

A general chat assistant can earn its place later, once you know what people actually ask. It's almost never the right first feature.

Design for being wrong

Every AI feature produces bad output sometimes, so treat that as a normal code path, not an edge case.

• Suggestions should look like suggestions. Pre-filled fields stay visually distinct until a human confirms. No generated content silently becomes a factual record.
• Irreversible actions always go through a human. Reading and suggesting are low-risk. Sending, deleting, paying, or modifying someone else's data is a different product with different rules.
• Give users a one-click "this is wrong." It costs almost nothing and turns every failure into a labeled sample.

Cost scales with usage, not seats

An ordinary feature costs roughly the same whether a customer uses it once a month or all day. AI features aren't like that. Every summary and every draft is a billed call, and your heaviest users can cost many times more than your lightest.

Before launch, do the math on realistic monthly usage per active user. If the number looks bad, the fix is architectural: generate a summary once when a record changes rather than on every page view; send only the context the task needs; route narrow tasks to smaller models.

The difference between usage-based and seat-based billing. Traditional SaaS mostly charges by seat, where a customer's marginal cost stays roughly flat whether they use it a lot or a little, so pricing and margins can be worked out in advance. AI features break that model: every summary and every draft is a real billed model call, and heavy users can cost several times what light users do. That means gross margin fluctuates with usage intensity rather than customer count. The usual response is architectural: generate a summary once when a record changes and cache it, rather than regenerating on every page view; send only the context the task truly needs to the model, avoiding dumping whole tables or databases; and route narrow tasks (like classification and routing) to smaller, cheaper models, saving the large models for situations that genuinely need complex reasoning.

Ship it like any risky feature

1. Build a small eval set first. Take 30 to 50 real examples from your own product, each with a written definition of what a good output looks like. Run it on every prompt change or model swap, so you can distinguish improvement from regression.
2. Launch behind a flag: internal users, then a few friendly customers, then everyone.
3. Measure the task, not the clicks. Did task time go down? Do users accept suggestions or rewrite them? Are they still using it a month later? Opening a panel once is curiosity, not adoption.
4. Keep watching after launch. Vendors update models under your feet, and your data drifts.

It also pays to route all model calls through one internal interface and keep prompts in config. Then trying a cheaper or better model is a config change plus an eval run, not a rewrite.

What I'd do first

Pick one slow, repetitive task. Fetch clean context using the user's own permissions to feed the model. Place the output exactly where the user already works, let it fail quietly, and then measure whether it sticks. One feature people keep using beats five they tried once.

One-line Summary

Adding AI to a mature product is less about choosing a model and more about task, permissions, placement, and cost.

Paragraph Summary

Most "add AI to the product" requests are really about adding a feature to a mature product with existing paying users and years of data structures, not building a brand-new AI product. The first version is usually a bottom-right chat assistant that demos beautifully but sees usage collapse fast, because users open the product to get work done, not to chat. The right starting point is finding slow, repetitive, error-prone existing tasks: summarizing long content, filling forms from messy information, drafting first versions, classifying and routing, and intent-aware search. The real engineering is around the model: assembling context, respecting existing permission systems, deciding which screen the output lands on, handling vendor failures, and estimating cost by usage rather than seats. Ship it like any risky feature: build an eval set, roll out behind a flag, measure task completion rather than clicks, and keep watching after launch.