Zepto Builds AI Customer Service with MLflow: An Evaluation-Driven Practice Guide

The Customer Service Challenge of Rapid Growth

Zepto, a rapidly rising instant delivery platform in India, faces pressure from massive customer inquiries. Quick Commerce is a retail model that has emerged in recent years, promising to deliver goods to consumers within 10-30 minutes. This model places extremely high demands on customer service systems: order status inquiries, delivery delay complaints, out-of-stock product replacements, and other issues require immediate responses. Traditional ticketing systems or 24-hour reply mechanisms cannot meet these needs at all—any delay could lead to user churn. The uniqueness of the Indian market also lies in its multilingual environment (Hindi, Tamil, etc.) and price sensitivity, making customer service systems need to be both efficient and cost-controlled.

With business expansion, traditional human customer service cannot meet real-time response demands. How to improve efficiency with AI technology while ensuring service quality has become the core challenge. Unlike many companies pursuing deployment speed, Zepto chose an "evaluation-first" technical approach—establishing a comprehensive evaluation system before system launch, allowing each iteration of the AI Agent to be quantitatively measured, finding a balance between reliability and performance.

Technical Architecture Design

Platform and Tool Selection

Zepto's AI customer service system is built on the Databricks platform, fully leveraging its data lake and computing capabilities. Databricks is a unified data analytics platform based on Apache Spark, originally created by the Spark founding team. Its core advantage is deeply integrating Data Lakehouse architecture with machine learning workflows. The platform provides Delta Lake storage engine for ACID transaction support, Photon query engine to accelerate data processing, and auto-scaling compute clusters. For AI applications, Databricks' unique value lies in connecting the entire pipeline from data preparation, feature engineering, model training to inference services, avoiding the overhead of moving data between multiple systems in traditional architectures.

MLflow serves as the model management core, spanning the entire model lifecycle—from experiment tracking and model registration to production deployment. MLflow is an open-source machine learning lifecycle management platform with four core components: Tracking records parameters, metrics, and artifacts for each training run; Projects define reproducible runtime environments; Models provide standardized model packaging formats supporting multi-framework deployment; Registry enables model version control and stage management (dev/staging/production). In AI Agent development, MLflow solves a critical pain point: researchers may try hundreds of prompt and parameter combinations—without systematic tracking, they fall into the chaos of "which version works best."

The core advantage of this selection: data processing, model training, and inference services are completed on a unified platform, significantly reducing system complexity. Enterprises don't need to maintain separate data warehouses, training clusters, and inference services, substantially lowering infrastructure complexity. Databricks' distributed computing capabilities enable teams to quickly process massive historical conversation data and extract high-quality training samples.

Evaluation-Driven Workflow

"Evaluation-first" is reflected in every development stage. During model selection, Zepto doesn't directly adopt the model with the most parameters, but designs multi-dimensional evaluation metrics for business scenarios: response accuracy, intent recognition precision, conversation completion rate, and average processing time.

Each candidate model is comprehensively tested on real customer service datasets. MLflow's experiment tracking records detailed parameters and metrics for each experiment, enabling teams to systematically compare the effectiveness of different models and prompting strategies. MLflow's experiment comparison interface allows teams to intuitively compare different configurations, while the model registry ensures production environments always run validated versions, avoiding accidental deployment of experimental code. This data-driven decision-making approach avoids subjective judgment biases.

Key Practices from Experiment to Production

Building High-Quality Evaluation Datasets

The foundation of the evaluation system is high-quality annotated data. Zepto selected thousands of typical cases from historical customer service conversations, covering common scenarios like order inquiries, refund requests, and delivery issues, with each sample manually reviewed to ensure accuracy.

This process lays the foundation for model optimization. With a standardized evaluation set, teams can quickly verify whether model or prompt changes truly improve performance, rather than just performing better on specific cases.

Automated and Human Collaboration Mechanism

AI Agents don't completely replace humans but establish a tiered processing mechanism. The system routes based on confidence scores, automatically handling high-confidence simple questions while transferring low-confidence or complex issues to humans. Confidence scoring is the AI system's internal mechanism for judging the reliability of its own answers. When large language models generate responses, they calculate probability distributions for each output token. By aggregating these probabilities (such as using average log probability or entropy values), an overall confidence score can be obtained. High confidence (e.g., >0.85) usually means the model has seen many similar cases in training data and the answer is more reliable; low confidence indicates the question may be beyond the model's capability range or has ambiguity. In practice, enterprises set thresholds based on business error tolerance: factual questions like order inquiries can have higher automation thresholds, while scenarios requiring judgment like refund decisions conservatively transfer to humans. This mechanism essentially lets AI "know what it doesn't know," avoiding negative impacts from generating incorrect information (hallucinations).

MLflow's model monitoring capabilities track this ratio in real-time, balancing automation rate and service quality. In production, Zepto continuously collects user feedback and human correction data, regularly updating evaluation datasets. This closed-loop feedback allows the AI system to continuously learn new patterns and maintain long-term performance stability.

Business Results and Experience Summary

Quantifiable Value Improvement

Through the evaluation-driven AI customer service system, Zepto achieved significant results: 60% improvement in customer service response speed, 40% reduction in manual handling volume, while maintaining high customer satisfaction. More importantly, the system's scalability allows Zepto to quickly allocate resources during business peak periods without large-scale hiring.

MLflow's model version management capabilities support safe A/B testing. A/B testing is a commonly used experimental method in internet products and is equally critical in AI model deployment. The specific approach is: split online traffic proportionally, such as 90% using the current production model (control group) and 10% using the new model (experimental group), evaluating the new model's effectiveness by comparing key metrics between the two groups (response accuracy, user satisfaction, human transfer rate, etc.). Tools like MLflow can track real-time performance of different model versions—if experimental group metrics decline, rollback can occur immediately. New models are validated with small traffic before going live, and only rolled out fully after confirmation. Compared to direct full replacement, A/B testing advantages include: validation under real traffic, discovering edge cases laboratory testing can't cover; limiting impact scope—even if the new model has issues, only a small portion of users are affected; making decisions after accumulating sufficient sample size, avoiding intuitive judgment. This progressive deployment strategy significantly reduces production incident risk and is standard practice in large-scale AI systems.

Replicable Practice Methodology

Zepto's practice proves that "evaluation-first" is not just a technical philosophy but a product mindset. In the rapidly iterating AI application environment, systematic evaluation capability determines whether enterprises can obtain sustainable value from technical investments.

For enterprises hoping to build similar systems, key insights include: prioritize investing in evaluation infrastructure rather than rushing to launch; choose platforms that unify data and model management to reduce complexity; establish a complete toolchain from experiment to production; continuously focus on real business scenarios rather than blindly pursuing technical advancement.

Directions for Continuous Evolution

As large language model technology evolves, Zepto's AI customer service system still has optimization space. Introduction of multimodal capabilities (such as image recognition for order issues), more refined sentiment analysis, and cross-language support are all potential directions.

But regardless of how technology changes, the core principle of evaluation-driven will continue to guide system evolution. Only AI applications built on reliable evaluation can truly deliver value in production environments. Zepto's experience provides the industry with replicable, verifiable best practice examples.

Key Takeaways

• Evaluation-first philosophy: Establish comprehensive evaluation systems before deployment rather than rushing to launch
• Unified platform architecture: Databricks + MLflow reduces complexity across data processing, model training, and inference
• High-quality evaluation datasets: Foundation for systematic model optimization and iteration
• Human-AI collaboration: Confidence-based routing balances automation efficiency and service quality
• Closed-loop feedback mechanism: Continuous collection of production data drives long-term system improvement
• Safe deployment practices: A/B testing and version management minimize production risks
• Business value validation: 60% response speed improvement, 40% reduction in manual handling demonstrate measurable ROI