AI Quality Inspection for 400K Tickets: An Intern's Architecture Decisions

One Intern Carrying an Entire AI QA Project

On Reddit, an intern at an IT support company shared a representative engineering challenge: the company handed him a large-scale AI project to complete almost entirely on his own. His resources: roughly 400,000 historical customer tech support records in SQL Server, each containing the customer's issue, the steps taken by support staff to troubleshoot or resolve it, and a QA team review.

His goal was to build an AI assistant — given a new issue and the handling actions logged by an employee, the system should provide scoring suggestions across three dimensions:

• Whether prescribed troubleshooting steps were followed → +2 / -2
• Whether the comments/actions are relevant to the issue → +2 / -2
• Whether the written comments are grammatically correct → +2 / -2

The constraints were realistic: QA staff still make the final call (AI is advisory only), the timeline is 1–2 months, the budget is limited, and CPU vs. GPU costs need to be weighed carefully. This isn't an academic pursuit of SOTA — it's a practical system that has to ship under real engineering constraints.

Breaking the Problem into Three Independent Subtasks

The biggest trap here is treating this as a "one big model handles everything" problem. In reality, the three scoring dimensions have vastly different technical characteristics — lumping them together makes the architecture fragile. The smarter approach is divide and conquer.

Grammar Checking: The Easy Win

Checking the grammatical correctness of written comments requires almost no model training at all. Mature rule-based engines and lightweight tools like LanguageTool can handle this on CPU with minimal latency and zero GPU cost. Pulling this dimension out of a complex ML pipeline immediately eliminates one-third of the engineering burden. It's the highest-ROI component of the entire project and should be shipped first.

Relevance Detection: Where Embeddings Shine

"Are the employee's actions relevant to the issue?" is fundamentally a semantic similarity/relevance problem. Encoding the issue description and the handling actions separately using a sentence embedding model, computing their semantic relatedness, and then training a lightweight classifier (logistic regression or gradient boosting) to output a +2/-2 judgment is a well-balanced approach in terms of both cost and performance.

The 400,000 historical records are enormously valuable here — they serve both as a source of labeled signals (historical QA pass/reject records) and as a retrieval corpus for similar cases. The embedding model can be a small open-source model that runs on CPU; offline batch encoding stores vectors in a vector database, keeping online query latency manageable.

Sentence embedding models map text of arbitrary length into a fixed-dimension dense vector space, where semantically similar texts are placed closer together. Unlike bag-of-words or TF-IDF, embeddings capture word order and semantic relationships — for example, "can't connect to network" and "no internet" would be mapped to nearby positions. Popular lightweight open-source options include all-MiniLM-L6-v2 (384 dimensions, ~80MB model size) and paraphrase-multilingual-MiniLM-L12-v2 (multilingual support), both capable of encoding a single entry in milliseconds on a standard CPU. Batch-encoding all 400,000 historical records offline and storing them in a vector database (e.g., FAISS or ChromaDB) means online requests only need to encode the new text and run one vector search — overall inference cost stays very low. A logistic regression or gradient boosting classifier then uses historical QA pass/reject labels as supervision, taking the concatenation or difference of the issue vector and action vector as input features and outputting a binary classification mapped to a +2/-2 score.

The Hardest Part: Troubleshooting Step Compliance

Among the three dimensions, "did the employee follow prescribed troubleshooting steps" is the real challenge — and the one the original poster struggled with most. The reason: it requires not just semantic understanding, but structured knowledge of what the "standard process" actually is.

Why a Pure Classifier Falls Short

Different issue types correspond to different standard troubleshooting steps. If you rely on a classifier to learn this purely from text, the model has to simultaneously identify the issue type and assess step completeness — a recipe for failure on long-tail edge cases with imbalanced data. A more robust approach is to introduce retrieval augmentation (RAG): for each new issue, retrieve the most similar historical cases that were previously judged by QA as "steps complete," and use those as reference standards.

Where RAG + LLM Fits In

This is where LLMs can add genuine value — but only if applied precisely. Assemble a prompt from the retrieved reference cases, the prescribed steps, and the employee's actual actions, and ask the LLM to judge whether the actual actions cover the required key steps. LLMs excel at this kind of contextual, comparative reasoning — something pure embedding similarity can't do.

That said, watch out for cost creep: LLM inference, whether via local GPU deployment or API calls, is the most expensive component in the entire system. Given the 1–2 month timeline and limited budget, the recommendation is to reserve LLM usage exclusively for the troubleshooting compliance dimension and use cheaper solutions for the other two.

Retrieval-Augmented Generation (RAG) is an architectural pattern that combines an external knowledge base with a language model: first, vector retrieval finds the most relevant document fragments from the knowledge base for the current input; then those fragments are injected into the prompt as context for the LLM to reference when generating a response. RAG's core value is that it frees the LLM from having to memorize all domain knowledge in its parameters — for this project, troubleshooting step standards will evolve with the business, so injecting the latest reference cases directly into the prompt is far more efficient than fine-tuning the model each time. For local deployment, open-source models under 7B parameters (e.g., Llama 3, Mistral 7B) with 4-bit quantization can run on a single consumer GPU or high-performance CPU, making them suitable for budget-constrained prototyping. To avoid GPU investment entirely, you can start with pay-per-use OpenAI API or Claude API calls, billing only for the troubleshooting compliance dimension and keeping API usage within a predictable range.

Recommended Architecture: A Layered Combination, Not a Silver Bullet

Taken together, the best fit for this scenario isn't any single technology — it's a layered combination:

1. Grammar dimension: Rule engine / LanguageTool, CPU, near-zero cost.
2. Relevance dimension: Sentence embedding + lightweight classifier, CPU-only, trained on historical QA labels.
3. Step compliance dimension: RAG to retrieve similar high-quality cases + LLM for step-by-step comparative reasoning — the only component worth spending GPU/API budget on.
4. Unified output layer: Each of the three modules produces a +2/-2 score, which are aggregated into a single recommendation for the QA team to make the final call.

On CPU vs. GPU

Since QA staff are the ultimate decision-makers and the system is purely advisory, real-time latency requirements aren't that strict. Embedding and classification can run entirely offline or in async batch mode — CPU is sufficient. Only the LLM step requires considering GPU, and even there you can first validate feasibility using a small open-source model on CPU before deciding whether to invest in a GPU or external API.

Practical Advice for a Solo Intern

This project is substantial for a single intern, but it's not unmanageable once decomposed. The key is not to be intimidated by the "large-scale AI" label, and not to jump straight into an end-to-end LLM approach.

Spend the first two weeks shipping a usable version of the grammar check and relevance detection — the "low-hanging fruit" — to build confidence and establish a baseline. Then focus the remaining time on the core challenge: troubleshooting step compliance. The real value of 400,000 historical records isn't training one massive model — it's serving as a labeling source and a retrieval reference corpus. That's the most economical use of the data asset.

Under real constraints of limited budget and limited headcount, "works, maintainable, cost-controlled" matters far more than "technically state-of-the-art." A system built from simple, composable modules — where every component is interpretable and debuggable — will earn more trust from the QA team than an opaque end-to-end black box.

For projects constrained by both time and manpower, establishing quantifiable evaluation metrics and staying aligned with the QA team is equally critical. Early in the project, sample a fixed test set from historical data (e.g., 500–1,000 records sampled evenly across issue types, with original QA judgments as ground truth) and evaluate accuracy, F1 score, and false positive rate on this set after each iteration. This has two benefits: it objectively measures whether each module improvement is genuine, preventing intuition-based parameter tuning; and it gives the QA team a readable "report card" that builds trust in the system. Also note that historical QA labels may contain noise (different QA staff applying inconsistent standards) — it's worth doing a quick label consistency analysis before training the classifier to avoid overfitting on noisy labels.