Put the Model Last: A Cheaper, Safer Architecture for Public Chatbots

A chatbot on a public website is essentially a paid endpoint that anyone can trigger. According to the Copper Larder case shared by developer Ahmadsn1 on their blog, treating the large model as the last step of a pipeline rather than the first can deliver a reliable user experience at extremely low cost.

Why Public Chatbots Are a Different Species

A chatbot behind a login faces known users and a limited audience. A bot hanging on a public website is an open endpoint, burning money with every message it receives. People will ask the same five questions all day long, try to get it to say something embarrassing, and occasionally paste in an entire novel.

Copper Larder is a demo guest-facing bot for a small UK pub, answering menu questions, taking callback requests, and streaming replies. What's really worth looking at is the pipeline wrapped around the model: a single request passes through 8 stages, and only stage 7 consumes tokens. Putting the cheapest, most deterministic checks first is the first principle of this design.

Open Endpoint refers to an API path that any unauthenticated person can access and trigger. Unlike internal interfaces that require login, an open endpoint has no user identity as a first filter, so both attack surface and cost risk are significantly higher. In a chatbot context, an open endpoint means every message actually invokes backend logic, including potentially triggering large model inference. Attackers can use scripts to send requests at extremely high frequency, causing a "Denial of Wallet" attack—not disrupting service availability, but draining your API quota and account balance through a flood of requests. This is exactly why Ahmadsn1 emphasizes lightweight gates up front: on an open endpoint, cost control must happen before requests reach expensive computation, and cannot rely on after-the-fact monitoring or rate limiting.

The Cheap Front Stages

Ahmadsn1's pipeline places several gates before the model: session caps, complaint escalation to a human, script interception, and exact-match caching.

• Session caps prevent a single visitor or script from blowing up the bill.
• Complaint escalation routes angry customers straight to a human, instead of hitting them with a cheerful canned response.
• Interception rules are pure pattern matching for high-frequency questions, like opening hours or whether there's parking. The pub has about 19 such rules, at zero token cost, giving the same correct answer every time.
• Exact-match caching returns a stored answer once the same normalized question has been answered before.

One detail worth praising: these answers come from structured restaurant data, not text hand-written into the rules. Menu cards and info cards are built only from type-defined data, so they can only display what the data itself permits. This avoids the classic problem of script rules drifting out of sync with real information.

const intercepts = [
  { test: /\b(opening|open)\s+(hours|times)\b|\bwhen are you open\b/i, answer: () => hoursCard(restaurant) },
  { test: /\bparking\b/i, answer: () => infoCard(restaurant.parking) },
]

export function tryIntercept(message) {
  const hit = intercepts.find((rule) => rule.test.test(message))
  return hit ? hit.answer() : null
}

Exact-Match Cache is the simplest caching strategy: it directly returns a stored result only when a new request, after normalization, is exactly identical to a previously answered question. It differs from semantic caching—the latter uses vector similarity to judge whether questions "mean roughly the same thing," but introduces extra embedding model costs and misjudgment risk. The advantage of exact-match caching is zero additional inference cost, strong determinism, return-on-hit, and it can be implemented entirely in ordinary code. In public chatbots, high-frequency repeated questions often make up a large share, so even exact matching alone can significantly reduce model calls. Normalization steps typically include trimming leading and trailing whitespace, unifying case, and collapsing consecutive whitespace, so that "What time do you open?" and "what time do you open" are recognized as the same question.

Remember What the Customer Said

If someone says "I'm vegetarian" in message 2, that constraint must still hold in message 20. Models forget, and long conversation histories get truncated. Copper Larder treats this as a session-wide dietary lock: the constraint is stored in session state, injected on every turn, and used to filter menu cards in code. The model is never trusted to remember an important constraint.

This approach generalizes to any constraint that needs to stay consistent across turns—allergens, language preferences, confirmed identity tiers—none of which should be left to the model's memory to maintain.

Check the Reply Before It Ships

The model is the only part of the pipeline that can say anything, so its output must be checked. In this bot, the critical rule is that it must never confirm availability, because it has no way of knowing. If a reply promises a table is free, the guardrail catches it, rewrites it with a correction, and—this is important—prevents that reply from entering the cache.

A cached mistake is a mistake you hand out for free to every future visitor.

const promisesAvailability = /\b(we have|there is|i can confirm|you are booked|table is available)\b/i

export function checkReply(reply) {
  if (promisesAvailability.test(reply)) {
    return {
      reply: reply + '\n\nI can’t see live availability, so please leave a callback request and the team will confirm.',
      cacheable: false,
    }
  }
  return { reply, cacheable: true }
}

Output Guardrails are checks executed after the model generates content and before it returns to the user. Unlike input-side interception rules, output guardrails deal with the fact that the model "can say anything." Common guardrails include: prohibiting promises about live state (such as inventory, availability, pricing), prohibiting personal information output, prohibiting legal or medical advice, and filtering inappropriate language. Guardrails are typically implemented via regex pattern matching or lightweight classifiers—extremely cheap but able to catch high-risk expressions. A key design principle: replies modified or blocked by guardrails should not enter the cache, because the purpose of caching is to reuse correct answers, not to distribute corrected "patch replies" as the standard answer to future users.

Always Have a Graceful Failure Ready

Models go down, get rate-limited, or in some environments keys are missing. Each of these paths returns a gentle, on-brand message and a callback card, so visitors can still leave a number. A broken demo is a bad impression; a polite "I'll have someone call you back" is just a slightly different product.

Two smaller habits to close with: replies are streamed via Server-Sent Events, paired with an aria live region so screen readers announce text as it arrives; and the app does not store raw IP addresses, so there's no sensitive data to leak.

The core insight of this architecture isn't complicated: most of the cost and security problems of public chatbots can be solved outside the model, with ordinary code. Put the model last, let cheap code answer what it can answer, refuse what it must refuse, cache what repeats, and only then hand the rest to the model—whose replies are checked once more before anyone sees them.

Originally published at ahmadhsn1.github.io.

Server-Sent Events (SSE) is an HTTP-based server push technology that allows a server to continuously send a stream of text events to a client over a single connection. In a chatbot context, SSE lets tokens generated by the model reach the user interface character by character, rather than waiting for the full reply to be generated before returning it all at once. This significantly improves perceived latency: users see text start appearing almost immediately, instead of staring at a blank screen for several seconds. SSE differs from WebSocket in that it is unidirectional (server to client), based on plain HTTP, and requires no additional protocol upgrade, making it easier to traverse corporate proxies and firewalls. Combined with an aria-live region, screen readers can automatically announce new text as it arrives, making streaming output equally accessible to users of assistive technology.

One-line Summary

Public chatbots should put the large model last in the pipeline, using ordinary code to solve most cost and security problems.

Paragraph Summary

A chatbot on a public website is an open endpoint where every message can burn money and introduce security risks. The Copper Larder case shows an 8-stage pipeline where only stage 7 calls the large model. Front stages handle high-frequency, deterministic questions with session caps, complaint escalation, script interception, and exact-match caching at zero token cost. Important session constraints like dietary restrictions are stored in state and injected every turn, not relying on model memory. Model outputs pass through guardrails before display, and corrected replies must not enter the cache, preventing mistakes from being distributed for free. When the model is unavailable, a brand-appropriate fallback message and callback card are returned. The core insight: most cost and security problems of public chatbots can be solved outside the model with ordinary code.