Building a Customer Support Copilot with Local Gemma 4: A Zero-Cost AI Assistant for a One-Person Shop

The customer support pain point of a one-person shop: repeating the same answers every night. A developer built the Little Kiln Support Copilot with local Gemma 4 and EmbeddingGemma, letting AI draft replies based on historical cases, running entirely on a laptop with zero API costs.

The Customer Support Dilemma of a One-Person Shop

For independent creators selling handmade goods on Etsy, Shopify, or their own sites, customer support is an invisible time sink. Take the fictional handmade ceramics shop Little Kiln Ceramics as an example: the owner throws pottery, takes photos, and packs orders during the day, then has to handle repetitive emails at night about "the mug broke," "the package is delayed," and "can I get a refund?" She knows the answer to every question, yet has to rephrase it each time to keep policies consistent.

She already knows the answer; she just needs to rewrite it consistently every time.

Existing solutions are either feature-heavy paid help desks that are too clunky for a one-person shop, or third-party AI APIs billed per message with uncontrollable long-term costs. Little Kiln Support Copilot targets exactly this gap: letting a local open-source model run on the shop owner's own laptop, charging a flat monthly fee or one-time installation fee, achieving a pricing model that is sustainable for both sides.

Core Architecture: How the RAG Pipeline Works

This Copilot is essentially a lightweight retrieval-augmented generation (RAG) system, written entirely in Python, relying on two open-weight models run locally via Ollama.

1. Vectorizing Historical Cases

Each historical case (customer message + resolution) is encoded with EmbeddingGemma (300M parameters, 768-dimensional vectors) and stored in a Postgres database, using the pgvector extension for vector retrieval. The author used Tiger Cloud's free tier for hosted database, but the code also supports local Postgres.

Vector embedding is the process of mapping a piece of text to a high-dimensional numerical vector, where texts with similar semantics are closer in the vector space. The 768-dimensional vector output by EmbeddingGemma is the "semantic fingerprint" of each case. pgvector is an open-source extension for PostgreSQL that adds a vector type and approximate nearest neighbor (ANN) indexing to the database, enabling "find the most similar records by cosine similarity" directly in SQL without introducing a separate vector database. For small-scale data like a one-person shop, the combination of Postgres plus pgvector is lightweight enough and easy to back up and migrate—this is the practical reason the author chose it.

2. Retrieving Similar Cases

After the new message is encoded the same way, a SQL query sorts historical cases by cosine similarity and keeps only entries above a threshold. This step determines the upper limit of subsequent generation quality.

Cosine similarity measures how close two vectors are in direction, typically ranging from -1 to 1, with values closer to 1 indicating greater semantic similarity. It only cares about direction, not vector length, making it suitable for comparing text embeddings. In practice, a similarity threshold is usually set: only retrieval results above the threshold are considered "relevant cases" and fed to the generation model. Too low a threshold brings noise into the context and induces fabrication; too high a threshold may miss cases that should match. The score boundary between 0.58 and 0.63 observed by the author is exactly the kind of empirical evidence worth recording when tuning parameters.

3. Local Draft Generation

Gemma 4 E2B (the quantization-aware e2b-it-qat build, about 4GB) receives the customer message and retrieved cases, with instructions to use only the policies in these cases and never fabricate prices or deadlines. The model runs on an ordinary laptop without a GPU.

RAG (Retrieval-Augmented Generation) is a technical architecture that combines information retrieval with text generation. Its core process is: first encode knowledge base content into vectors and build an index; when a user asks a question, the system retrieves the most relevant knowledge snippets and feeds them as context to the generation model, so the model answers based on these real materials rather than parameter memory alone. The key value of RAG lies in two points: first, it mitigates the "hallucination" of large models, making answers verifiable; second, it allows the model to access private or up-to-date domain knowledge without retraining. In this project, historical customer support cases are the retrieved knowledge base, and Gemma 4 plays the role of generator; retrieval quality and generation quality together determine the reliability of the final draft.

Quantization-Aware Training (QAT) is a method that simulates low-precision numerical operations (such as 4-bit or 8-bit integers) during training, so the model adapts to quantization errors during training and maintains good accuracy after being deployed as a compressed model. The "qat" in the model name refers to this. Compared with post-training quantization, QAT typically reduces accuracy loss, which is an important reason why the roughly 4GB Gemma 4 E2B can run on an ordinary laptop without a GPU while still producing usable drafts. For local deployment scenarios, quantization directly determines memory usage and inference speed, and is key to feasibility on small devices.

Three Key Lessons: From Errors to Reliability

The author discovered three issues in testing that directly affect product reliability, and solved them by adjusting parameters and prompts. These lessons have universal value for building similar tools.

Mismatched Policy: The Model Treats "Damage Refund" as "No-Reason Return"

The test message "The vase arrived broken, I want a refund" retrieved both the damage case and the no-reason return case, and Gemma sometimes chose the latter—accepting returns of unused items with the customer bearing shipping. This is obviously unsuitable for a customer holding a broken vase.

The fix has two parts: explicitly require in the prompt that the resolution of the most similar case be applied, and lower the temperature to 0.2. After adjustment, the draft correctly and consistently applies the damage policy: free replacement within 2 days, no need to return the broken item, provide photos of the damage.

Similarity Threshold: Low Thresholds Induce Fabricated Answers

With an initial threshold of 0.5, a new question about a teapot pulled in irrelevant cases, and Gemma confidently claimed the shop does not have teapots—information it could not know. Observation showed that relevant cases scored above 0.63 and noise below 0.58, so the threshold was raised to 0.6. When there are no relevant cases to reference, the model now honestly states that the shop owner needs to confirm, rather than fabricating an answer.

This adjustment reveals an important principle: the honesty of a RAG system depends not only on the model itself, but also on the filtering strategy of the retrieval stage.

Retrieval Is Not Resolution: The Generation Step Is Indispensable

The author had previously built only the search part and found that "I want a refund" could not find notes about defective shoes, because embeddings capture what the text is "about," not what "follows from it." After adding the generation step, the system could connect "broken" with "this is how we handle it." This illustrates the limitation of pure retrieval tools in customer support scenarios; the generation layer is the bridge for policy application.

Why Open-Source Local Models Matter for a One-Person Shop

The author explains the advantages of open models from both business and privacy perspectives, and these arguments are instructive for understanding the value of local AI in small business scenarios.

Zero marginal cost: Every draft is generated on the shop owner's laptop, so replying to 20 emails a day incurs no API fees, and a fixed low price is sustainable for both sides. With paid APIs, costs grow linearly with the number of customers.

Data privacy: Customer names, orders, and complaint content are never sent to third-party AI providers. Historical cases are stored in Postgres controlled by the author; for full privacy, one can switch to local Postgres plus pgvector.

Low hardware barrier: Gemma 4 E2B runs without a GPU, matching the devices independent creators actually own.

Multilingual for free: EmbeddingGemma supports over 100 languages, and the author previously tested the same setup with Italian notes. The shop can reply in the customer's native language without changing products.

Fully tunable: The model, temperature, threshold, and prompts are all under the shop owner's control, and these are exactly the key knobs for fixing wrong policies and fabricated answers.

The author also candidly acknowledges the trade-off: larger closed-source models may write more fluent text, but for drafts that require human review, a free, private, local model is the better value.

Product Positioning and Takeaways

Little Kiln Support Copilot is currently a command-line demo, demonstrating its capabilities through three real cases: a broken vase triggers the replacement policy, a delayed package references the SORRY10 discount code, and an uncovered question is honestly escalated to the shop owner. You may not have noticed that in the delayed package case the model leaked internal phrasing like "in the past we have offered...", which precisely proves the necessity of human review for every draft.

For a one-person shop, this project demonstrates a viable path: use local open-source models to handle repetitive customer support drafts, maintain business sustainability with a fixed fee, and keep customer data local. It is not meant to replace the shop owner, but to compress nightly repetitive labor into two actions: review and send.

The project code is open-sourced on GitHub, and the author submitted it to the "Side Hustle" track of Week 2 of the Hacktoberfest 2026 Open Source AI Challenge, competing for the "Best Use of Gemma" and "Best Use of Tiger Data" awards.

One-line Summary

Developer builds a RAG customer support Copilot with local Gemma 4 and EmbeddingGemma, with zero API costs and data never leaving the local machine.

Paragraph Summary

The article introduces Little Kiln Support Copilot, a local AI customer support drafting tool for one-person shops. Using a fictional handmade ceramics shop as an example, it points out that independent creators must repeatedly answer similar questions about breakage, delays, and refunds every night, while existing paid help desks are too heavy and third-party API costs are uncontrollable. The tool is written in Python and runs locally via Ollama, using EmbeddingGemma for vectorization, pgvector for similar case retrieval, and Gemma 4 E2B to generate reply drafts. The author summarizes three key lessons: prompts must require using the most similar case and lower temperature to fix policy mismatches; the similarity threshold should be raised to 0.6 to avoid fabricated answers; the generation step is a necessary link between retrieval and policy application. The article also explains the value of local open-source models in zero marginal cost, data privacy, low hardware barriers, multilingual support, and tunability, and emphasizes that drafts still require human review.