Auto-Scoring Every Support Call with AI: An Open-Source QA Scoring Solution Explained

The Blind Spots in Traditional Call Center QA

Most call center quality assurance (QA) processes audit only a tiny fraction of recorded calls. This sampling approach creates two persistent problems: compliance issues can easily slip through the cracks, and it's difficult to accurately assess whether an agent's performance is steadily improving. When a team handles hundreds or thousands of calls per day, evaluating overall performance based on a handful of manually reviewed recordings rarely yields statistically meaningful conclusions.

An open-source example project built on Telnyx Decision Models takes a different approach: rather than sampling, it automatically scores every completed call and continuously accumulates the results into each agent's quality profile. This full-coverage methodology fundamentally changes the data foundation for quality assurance.

Telnyx Decision Models is an AI inference service offered by the Telnyx communications platform. It allows developers to embed large language model (LLM) text understanding capabilities into workflows, enabling structured classification, scoring, or decision-making on input content. Unlike calling a general-purpose LLM API directly, Decision Models is designed for "judgment-type" tasks — developers can predefine scoring dimensions and output formats (such as a JSON schema), allowing the model's output to feed directly into downstream business logic without complex prompt engineering or output parsing. In the QA scenario described here, it handles the core task of receiving call transcripts and returning structured scoring results based on predefined quality and compliance standards, which are then used for metric accumulation and anomaly detection.

What This Solution Actually Does

According to the project author's post on Reddit, the system is built around "full-coverage scoring + continuous tracking" and delivers a complete set of capabilities:

• Score call transcripts against quality and compliance standards, rather than relying on surface-level metrics like call duration;
• Track a rolling five-call average score, evaluating agents based on trends rather than individual performance;
• Flag agents who may need coaching, so managers can direct their attention to those who genuinely need support;
• Immediately escalate serious compliance violations, preventing risks from being buried between sampling cycles;
• Store the complete scoring history in a SQL database, creating a traceable long-term record;
• Automatically send daily summary reports via SMS to team leads.

Taken together, these capabilities mean managers can receive a clear team quality summary every morning without having to manually listen through recordings.

Why the Rolling Average Matters

The "rolling five-call average" design is worth highlighting on its own. Scores for individual calls can easily be skewed by random factors like a customer's mood or the difficulty of an issue. Averaging across five consecutive calls smooths out the noise and more accurately reflects an agent's consistent performance level. It also transforms the question "is this agent improving?" from a subjective impression into a quantifiable trend.

Technical Highlight: Stateful Actors Preserve History

A key engineering decision in this solution is assigning each agent a dedicated Stateful Actor. Its role is to ensure that an agent's scoring history and scheduled reporting tasks survive system restarts — rather than vanishing between calls.

For scenarios that require building long-term quality profiles, state persistence is a hard requirement. If every service restart wipes the context, "long-term tracking" becomes impossible in practice. Through a combination of stateful design and SQL persistence, the system can build genuine, time-spanning agent profiles.

The project code is open-sourced on GitHub (under the post-call-qa-scoring directory in the team-telnyx/telnyx-code-examples repository). The underlying scoring capability relies on Telnyx's Decision Models inference service, and developers can reference the implementation directly or adapt it for their own use.

---

Stateful Actor is a concurrency programming concept rooted in the Actor Model. In the Actor Model, each "actor" is an independent unit of computation with its own private state, interacting with the outside world through message passing. "Stateful" means the unit retains its internal context across multiple message-handling cycles, rather than resetting after each invocation.

In Telnyx's implementation, each agent corresponds to a dedicated Stateful Actor instance that holds the agent's historical scores, current rolling average, and the scheduling state for periodic reports. When a new call score is written, the Actor directly updates its own state and triggers downstream logic (such as determining whether escalation is needed), without needing to query the database from scratch to rebuild context. The SQL database serves as the persistence layer, ensuring the Actor can recover to its last state snapshot after a process crash or service restart. The two layers form a complementary structure: fast in-memory reads and writes, backed by durable on-disk persistence.

Balancing Automation with Human Review

The author closes their post with an open question that sits at the heart of deploying solutions like this: How do you achieve automated QA without removing humans from high-stakes decisions?

This reflects a pragmatic stance. The value of automated scoring lies in its coverage — it brings 100% of calls under monitoring, eliminating the blind spots created by sampling. But when it comes to high-risk decisions like compliance penalties or agent performance evaluations, the model's judgment should serve as input, not a final verdict. A sensible division of labor is to let AI handle large-scale initial screening and anomaly flagging, while concentrating the valuable attention of human reviewers on serious cases that the system escalates.

In other words, automation isn't about replacing the QA team — it's about redistributing their attention, shifting it from mechanical spot-checking to deep judgment on critical cases.

Implications for the Industry

This open-source example is modest in scope, yet it illustrates a typical path for deploying generative AI in enterprise operations: rather than chasing flashy conversational capabilities, it wires LLM text understanding into a well-defined business process to solve the specific pain point of incomplete scoring coverage.

For call centers considering AI-powered QA, this solution offers a referenceable architectural template: transcript scoring, rolling metrics, state persistence, anomaly escalation, and scheduled reporting — each component maps to a clear operational need. More importantly, "preserving human review" is baked into the design philosophy from the start, not bolted on as an afterthought.