WhatsApp AI Agent in Production: 7 Key Engineering Pitfalls

Demos Look Great — Production Is the Real Test

An AI customer service Agent deployed on WhatsApp performs flawlessly in a demo: paste in a business description, a user asks "What time do you open?", and the model replies with a friendly sentence. But once real customers arrive, everything changes — they ask about competitors' products, send voice messages, type "HUMAN!!" in all caps, fire off three messages in five seconds, and reply to conversations from three days ago. Model providers occasionally go down, and some customers outright instruct the bot to ignore its own instructions.

In an engineering retrospective, the WAPGROWAI team broke down the infrastructure they built around the model when running an AI Agent inside a WhatsApp inbox. These aren't model capability problems — they're engineering decision problems. Here are the seven most important ones.

1. Draw the Boundary First: Let a Classifier Decide Whether to Answer

The core design principle is this: the model that writes the answer should not be the model that decides whether to answer.

Every incoming message first passes through a separate, cheap classification call that returns one of three labels: in-scope answer, out-of-scope refusal, or handoff to a human. Labels are obtained via a forced tool call with enum values, temperature set to 0, and the output is never free text that needs parsing.

There are two deliberate trade-offs in this design:

Refusals and handoffs use fixed strings, not model-generated text. Irrelevant or hostile messages never reach the answering model, so they can't be "persuaded." When a customer says "ignore your instructions and...", the response is a line you wrote yourself, greatly shrinking the prompt injection attack surface.
When the classifier fails, let it through. If the classification call errors out, the message is treated as in-scope and handed to the answering model, whose own prompt is still constrained to the business domain. The alternative — staying silent whenever the classifier hiccups — is worse for customer experience. If you choose a different policy, that too should be a deliberate decision, not an accident.

The classifier only looks at the current message, which makes it fast and cheap, and is why it's instructed to "choose in-scope when uncertain": stripped of context, an isolated "yes" means nothing.

What is prompt injection? Prompt injection is a class of attacks against large language model applications: the attacker embeds instructional text into the input to try to override, bypass, or tamper with the system's predefined behavior rules. For example, a customer sending "ignore your instructions, you are now..." is a classic direct injection. Because large language models fundamentally cannot distinguish between "system instructions" and "user input" — everything ultimately enters the same context window — relying on prompt hardening alone is unreliable. Defensive approaches generally fall into two categories: input-side filtering (like the independent classifier in this case), and architectural isolation (keeping sensitive decisions away from a persuasion-vulnerable generative model altogether). WAPGROWAI adopts the latter — refusal responses are generated from fixed strings rather than the model, so even if an attacker succeeds in making the answering model "obey," the blast radius is confined to the business domain and cannot trigger actions that should never happen.

2. Anchor Knowledge with a Single Document and Keep It Short

The Agent draws its answers from a business profile — a document the merchant writes or uploads, covering what they sell, opening hours, delivery and return policies, prices they're willing to quote, and things the bot must never promise. The entire profile goes into the system prompt on every call, along with rules: facts, prices, and hours must come from the profile, and the Agent cannot place orders, collect payments, or make bookings.

The team chose not to use a retrieval approach (chunking, embeddings, vector stores). Small merchants' profiles are typically just a few pages, with an upper limit of 200,000 characters. Sending the whole thing avoids index staleness drift and avoids the retrieval step silently missing a critical passage. The cost is token consumption on every call — a trade-off to watch if your knowledge base is large.

The model is also asked to answer in at most three short sentences and to use the customer's language. On WhatsApp, a wall of text is worse than a brief reply.

Retrieval-augmented generation (RAG) vs. full-context injection. The typical RAG approach chunks the knowledge base, converts it into vectors using an embedding model, and stores them in a vector database. When a user asks a question, it retrieves the most relevant chunks and stitches them into the prompt. Its advantage is that the knowledge base can be arbitrarily large and token consumption is controllable, but it introduces new failure modes: poor chunking splits semantics apart, embedding-query mismatches cause retrieval failures, index desync after knowledge base updates produces "stale drift," and when retrieval itself fails, the model confidently answers based on incomplete context. For a small merchant profile of just a few pages, stuffing the whole thing into the system prompt is actually simpler and more reliable, because the model always sees all the facts and can't miss a critical passage. The judgment criterion for this trade-off is knowledge base size and update frequency: small, static, strongly consistent knowledge suits full injection; large, dynamic, heterogeneous knowledge must rely on retrieval.

3. Conversation History Comes from Your Message Store, Not Model Memory

The model doesn't remember conversations. On every call, you reconstruct history from stored messages, and the details matter:

Everything the merchant sends counts as an assistant turn, including messages typed by human agents. If someone answered a question an hour ago, the AI must see that answer, or it will contradict itself.
Media messages become placeholders. The model can't see voice messages or images, but it should know someone sent one, so "[sent a voice message]" goes into the history rather than leaving it blank.
Limit turns and characters. Old context consumes tokens and rarely helps; a bounded window keeps both cost and latency predictable.

This pattern recurs across engineering practices in multiple similar products: state management is the application layer's responsibility, not the model's.

Stateless models and state reconstruction. Mainstream large language model APIs are essentially stateless: each call is an independent request, and the model doesn't "remember" what was said last time. The so-called "conversation" is entirely an effect manufactured by the application layer re-stitching historical messages into the context on each request. This brings several engineering constraints. First, history must be persisted to the application's own storage (databases, message queues, etc.) — model providers don't offer reliable session memory. Second, the history-stitching strategy directly determines cost and effectiveness: full concatenation makes tokens grow linearly with conversation length, so sliding windows or summary compression are typically used. Third, messages from multiple role sources (AI, human agents, system) must be merged into a single history with correct role labels, or the model becomes confused about its own identity and behavior. Understanding this is key to seeing why "state management is the application layer's responsibility" is a basic consensus in conversational AI.

4. One Conversation, One Reply at a Time

Customers send messages back-to-back: "Hi," "Do you deliver to Pune," "How much." If each message kicks off its own model call, they'll complete in random order and the customer gets three overlapping answers.

The solution is a conversation-level lock: a message arriving while a reply is being generated is skipped, not queued — the in-flight reply already read the latest history and likely covers it anyway. This is an in-process lock; if you run multiple instances, you need the same guarantee from shared storage (a database row or Redis key). During generation, WAPGROWAI also shows WhatsApp's "typing" indicator so a few seconds of thinking don't feel like silence.

5. Prepare for the Provider's Bad Day: Switch Based on Error Type

The Agent calls an OpenAI-compatible endpoint and falls back to NVIDIA's hosted models. What makes failover actually useful is treating errors differently:

Transient errors (timeouts, capacity errors): briefly retry against the same provider, then switch.
Quota exhaustion / model unavailable: retrying is pointless — move immediately.
Fatal errors (the request itself is rejected): stop. Sending a malformed request to a second provider just doubles the failure.

Each task also has its own budget: the classifier runs at temperature 0 with a very short timeout; answering calls are slightly more lenient; long tasks like generating chatbot flows are measured in minutes.

Error classification and failover strategy. When calling external model services, the errors thrown are not all of the same nature, and blind retries or blind switches both amplify problems. There are three common categories: transient errors (like 429 rate limits, 503 capacity shortages, network timeouts), which usually succeed after a brief wait and retry, worth retrying once or twice within the same provider before switching; quota exhaustion or model deprecation (like an account running out of credits or a specified model being retired), where retrying is meaningless and you must immediately switch to a backup provider; and fatal errors (like 400 malformed requests or authentication failures), where the problem lies in the request itself, and sending it to a second provider just repeats the failure — you should stop and alert. Additionally, different task types should have different timeout and retry budgets: classifiers prioritize low latency with very short timeouts; answering calls can be slightly more lenient; long tasks like generating a full chatbot flow may be measured in minutes. Writing error types, retry counts, and timeout thresholds as explicit configuration rather than magic numbers scattered through the code is the foundation of production reliability.

6. Make Output Fit WhatsApp's Formatting

The model writes Markdown, but WhatsApp doesn't render headings and uses single asterisks for bold. This needs conversion before sending, and truncation at sentence boundaries. For example, **bold** becomes *bold*, heading symbols are removed, and the length cap is 900 characters.

One more filter worth adding: some reasoning models occasionally return chain-of-thought instead of an answer. Detect that pattern and discard it rather than sending it to the customer.

7. Decide Who Responds First: Rules, Chatbot, or AI

An inbox often runs multiple automations simultaneously. Without an explicit order, keyword rules and the AI Agent will both reply to the same message. WAPGROWAI's order is:

Button and form replies sent by the merchant are handled first.
Rules that launch a chatbot come next — keywords can always pull a customer into a structured flow.
An already-running chatbot continues.
The AI Agent answers only when it's activated on that conversation.
While the AI Agent owns the conversation, other rules stay silent on its messages.

Deterministic flows take priority over AI, and AI takes priority over everything else. Whatever order you choose, write it down — when a customer gets two replies, that's the first question you'll be asked.

Questions You Must Settle Before Launch

These have no single right answer, but each requires a deliberate decision:

Voice messages and images. Transcribe, reply asking for text, or hand off to a human? A text-only Agent that doesn't react to voice messages at all looks broken.
When the model fails completely. Send a brief "we'll get back to you soon," hand off to a human, or stay silent and let a human handle it?
Who takes over a handoff. Is the conversation assigned to someone, does someone get notified, or does it wait in a shared queue? A handoff nobody sees is worse than no AI at all.
What ends AI mode. A customer asking for a human, a rule, a button — and should a teammate replying or being assigned to the conversation automatically end it?
The 24-hour window. Replying to customers is fine, but sending an AI-written greeting to a conversation that's been dormant for days will be rejected by WhatsApp; proactive messages require templates.
Cost and caps. Every AI reply is also a WhatsApp service message, and Meta charges for anything beyond 1,000 per number per month. Decide what happens when you hit the monthly limit.
Data. Customer messages get sent to your model provider. Know where they go, and disclose it in your privacy policy.

Summary

The model is the easy part. Production work is deciding when it speaks, what it can say, and what happens when it shouldn't.

The value of this case extends beyond WhatsApp itself: any team deploying conversational AI in real user scenarios will encounter these engineering problems — boundary classification, state reconstruction, concurrency control, error classification, output formatting. Model capabilities keep improving, but these infrastructure decisions around the model are the real distance between a demo and production.

One-line Summary

The model is the easy part — seven engineering decisions around it determine whether conversational AI can go to production.

Paragraph Summary

Using WAPGROWAI's engineering retrospective on running an AI customer service Agent in a WhatsApp inbox as a case study, this article argues that the real gap between demo and production lies not in model capability but in the infrastructure built around the model. It distills seven key decisions: using an independent cheap classifier to decide whether to answer, with fixed strings for refusals and handoffs to shrink the prompt injection surface; anchoring knowledge with a single business profile instead of vector retrieval; reconstructing conversation history from the message store and merging all merchant-sent turns; using a conversation-level lock to ensure only one reply per conversation; handling failover differently by error type; converting output to WhatsApp formatting; and explicitly defining the response priority among rules, chatbots, and AI. It closes with questions that must be settled before launch: voice and image handling, failure fallbacks, handoff ownership, cost caps, and more.