Low-Cost Abstention Strategy: How LLM Ticket Classification Reduces Misrouting Costs

Automatically routing IT support tickets to the wrong team can cost more than manual triage. A new study proposes a "Cost-Aware Abstention" framework that quantifies the performance of different models under explicit costs by selecting a routing strategy on calibration data and evaluating it once on a test set. The study compared four traditional classifiers and four large language models on a public dataset of 47,837 tickets across eight categories, and found that TF-IDF with logistic regression at a threshold of 0.60 reduced test cost from 0.763 to 0.472—a relative reduction of 38.1%—while lowering the misrouting rate from 15.3% to 3.5%.

Why Cost-Aware Abstention Is Needed

In IT operations, ticket misrouting means the issue is forwarded to the wrong expert, which not only delays resolution but can also lead to repeated communication and user dissatisfaction. The study sets an illustrative cost model: (c_ok, c_triage, c_wrong) = (0, 1, 5), meaning the cost of correct routing is 0, manual triage costs 1, and misrouting costs 5. This setting reflects the reality that misrouting is far more costly than human intervention.

Traditional classifiers typically pursue overall accuracy, but high accuracy does not equate to low cost. For example, a model may correctly classify a large number of tickets, but a few high-risk errors incur disproportionate costs. Cost-aware abstention allows the model to "abstain" when uncertain, handing the ticket to manual triage and thereby avoiding costly errors.

The core idea of the study is: select a routing strategy (including the abstention threshold) on calibration data based on explicit costs, then freeze that strategy and evaluate it once on a held-out test set. This simulates the offline evaluation scenario before real deployment and avoids the optimistic bias introduced by tuning parameters on the test set.

Abstention in machine learning, also known as "reject prediction" or "selective prediction," refers to a model actively declining to give a classification result when its confidence is insufficient, handing the sample to human processing. This mechanism stems from risk minimization theory: when the expected risk of the model's prediction exceeds the cost of manual processing, abstention is the better choice. The abstention threshold is usually set based on calibrated confidence or probability and can be viewed as a trade-off knob between coverage and accuracy. In the ticket routing scenario, abstention means forwarding the ticket to a manual triage desk rather than forcibly assigning it to an expert queue.

Experimental Setup: Traditional Models vs. Large Language Models

The study used four trained non-LLM classifiers (including TF-IDF+logistic regression, linear SVM, etc.) and four large language models (including GPT-4o-mini, Claude Haiku, etc.), compared under zero-shot and few-shot prompting. The dataset contains 47,837 tickets divided into eight categories.

In addition to conventional accuracy, the evaluation metrics include cost, misrouting rate, correctness AUROC, and risk-coverage area. In particular, the study notes that isotonic calibration can reduce expected calibration error (ECE) but does not always improve discrimination. For example, the GPT-4o-mini few-shot configuration had only one confidence value before and after calibration, with a correctness AUROC of only 0.500, indicating that the model has almost no ability to distinguish correct from incorrect predictions.

Therefore, the study recommends reporting both correctness AUROC and a tie-aware risk-coverage area to more comprehensively evaluate the quality of model uncertainty.

Correctness AUROC measures a model's ability to distinguish "correct predictions" from "incorrect predictions," with 0.5 indicating no discrimination and 1.0 indicating perfect discrimination. It is threshold-independent and is a threshold-free metric for evaluating uncertainty quality. Risk-Coverage AUC characterizes the area under the error rate curve as the model abstains on different proportions of samples, where coverage is the proportion of routed samples and risk is the error rate. When multiple samples have the same confidence (ties), this metric requires special handling. Combining both avoids the blind spot of looking only at calibration error (ECE): a model may be well-calibrated (predicted probabilities match actual correctness rates) but have identical confidence for all samples, making effective abstention via threshold impossible.

Key Findings: Traditional Methods Outperform LLMs

Under the illustrative cost model, TF-IDF+logistic regression with a threshold of 0.60 selected on the calibration set reduced test cost from 0.763 to 0.472, a relative reduction of 38.1% (paired bootstrap 95% CI [35.7%, 40.4%]), and lowered the misrouting rate from 15.3% to 3.5%. This result significantly outperforms all tested LLM configurations.

On a stratified sample of 1,000 test cases, the strongest prompt configuration achieved an accuracy of 0.477, while a fine-tuned DistilBERT reached 0.879. This indicates that for ticket classification tasks, a fully trained small model can be more cost-effective and accurate than large language models.

The study also conducted a post-hoc cost parameter analysis and found that the frozen calibration-selected strategy outperformed the "always route" and "always abstain" baselines in 14 out of 16 settings. This demonstrates the robustness of the cost-aware abstention framework under different cost parameters.

Paired bootstrap is a non-parametric statistical method used to estimate confidence intervals for the ratio of two related metrics. In this study, it resamples the same set of test samples with replacement, computes the ratio of "abstention strategy cost" to "always route cost" in each resample, repeats thousands of times, and takes the 2.5% and 97.5% quantiles as the 95% confidence interval. This method does not assume that cost differences follow a normal distribution and preserves the paired relationships between samples, making it suitable for assessing uncertainty in ratio-type metrics such as relative reduction.

Practical Impact of the Abstention Threshold

At the analytically derived break-even probability of 0.80, seven prompt configurations chose to abstain on all tickets, while Haiku few-shot routed only 19 out of 1,000. This means that when model confidence is below 0.80, abstaining is more economical than risking routing. This threshold can be adjusted according to the actual cost structure, providing flexibility for deployment.

The study also compared marginal/entropy-based rejection baselines and LLM-only consistency controls, further distinguishing the roles of calibration and discrimination. The results show that relying solely on LLM confidence or consistency may be insufficient to achieve cost-optimal abstention.

Break-even probability refers to the critical value at which the probability of a correct model prediction exactly equalizes the expected costs of "routing" and "abstaining." Under the cost model (c_ok, c_triage, c_wrong)=(0,1,5), let the model confidence be p; the expected cost of routing is (1-p)×5, and the cost of abstention is 1. Setting them equal gives p=0.8. When p>0.8, routing is more economical; when p<0.8, abstention is better. This derivation assumes that confidence is well-calibrated, so calibration quality directly affects the reliability of the break-even threshold. In actual deployment, changes in cost parameters will shift this threshold, and organizations need to recalculate based on their own misrouting costs and labor costs.

Limitations and Deployment Implications

The authors explicitly state that the study is limited to a specific dataset, prompt design, model endpoints, and illustrative costs. This work is an evaluation framework, not a new abstention algorithm or a general deployment certification. In actual deployment, organizations need to recalibrate the threshold according to their own cost structure and consider changes in ticket distribution.

For IT operations teams, this study conveys a pragmatic signal: for structured tasks like ticket classification, traditional machine learning models combined with cost-aware abstention may be more economical and reliable than directly calling large language models. LLMs are not a panacea, especially in scenarios requiring precise cost control; lightweight models plus calibration strategies remain a solution worth prioritizing.

Conclusion

Cost-aware abstention provides a method for offline evaluation readiness of LLM ticket classification. Through an explicit cost model and calibration-set strategy selection, teams can quantify the trade-off between misrouting risk and manual triage cost before deployment. Experiments show that TF-IDF+logistic regression significantly outperforms multiple LLM configurations on cost metrics, with a substantial reduction in the misrouting rate. This framework emphasizes the separate evaluation of calibration and discrimination, offering actionable guidance for real-world deployment.

One-line Summary

A cost-aware abstention framework shows TF-IDF+logistic regression reduces ticket routing costs by 38.1%, outperforming multiple LLM configurations.

Paragraph Summary

A new study proposes a cost-aware abstention framework for offline evaluation of IT ticket classification models under explicit costs. The study compares four traditional classifiers and four large language models on a dataset of 47,837 tickets across eight categories, using an illustrative cost model (0,1,5). The core method selects an abstention threshold on a calibration set, freezes it, and evaluates once on a test set. Results show that TF-IDF+logistic regression at a threshold of 0.60 reduces test cost from 0.763 to 0.472—a relative reduction of 38.1%—and lowers the misrouting rate from 15.3% to 3.5%, significantly outperforming all LLM configurations. A fine-tuned DistilBERT achieves an accuracy of 0.879, far exceeding the strongest LLM prompt's 0.477. The study emphasizes the need to report both correctness AUROC and risk-coverage area to assess uncertainty quality, and notes that the framework is an offline evaluation tool; actual deployment requires recalibrating the threshold based on the organization's cost structure.