Shopify Agent Cold Start in Practice: A Three-Step Method for Training Production-Grade AI with Zero Conversation Data

When an enterprise-grade Agent enters production, the margin for error is extremely low—especially for Agents handling business processes. It's not dealing with casual chat, but rather the rules, conditions, actions, and automated workflows in the backend. A single incorrect tool call could trigger the wrong business rule, causing real financial losses.

Shopify recently published a detailed technical blog describing how they brought a production-grade AI Agent from zero to deployment. The most noteworthy aspect is their approach to solving the "cold start" problem—how to construct high-quality training samples when there's no real user conversation data available.

Business Context: Having an AI Assistant Auto-Generate E-commerce Workflows

Shopify has a product called Shopify Flow, which is essentially an automation workflow tool for e-commerce backends. Merchants can use it to set up various rules: automatically alert the team about high-risk orders, send notifications when inventory is low, automatically tag customers after they complete certain actions, send emails, or trigger follow-up processes.

Flow is a concrete implementation of iPaaS (Integration Platform as a Service) and RPA (Robotic Process Automation) in the e-commerce vertical. In e-commerce operations, merchants face a large volume of repetitive decisions daily: inventory management, order risk control, customer segmentation, marketing outreach, and more. Traditional approaches rely on manual operations or developers writing code, while Flow uses a visual Trigger-Condition-Action paradigm that allows non-technical users to build automation rules. This pattern has been validated in general-purpose automation platforms like Zapier and Make, but Shopify's advantage lies in deep integration with its own ecosystem—including hundreds of native data objects and events such as orders, inventory, customers, and discounts—allowing automation rules to directly operate on core business entities rather than merely performing surface-level data transfer.

Now, Shopify wants its AI assistant Sidekick (internal codename Cyclone) to directly help merchants generate these Flows. A merchant just needs to say one sentence in natural language, and Cyclone needs to output a complete automation workflow executable in the backend.

The difficulty lies in Tool Call complexity: the model needs to know when to search for tools, when to select triggers, when to add conditions, when to configure actions, and finally how to assemble these components into a workflow that Shopify's backend can execute. This isn't a simple Q&A task—it's a multi-step, strongly-constrained tool calling chain.

Tool Call is the core capability of current AI Agent architectures, originating from the Function Calling mechanism promoted by OpenAI in 2023. Its essence is enabling large language models to output structured function call instructions—including function names and parameters—in addition to generating natural language responses. The model itself doesn't execute these functions; instead, an external system receives the instructions, executes them, and returns the results to the model for the next round of reasoning. In Shopify's scenario, Tool Call complexity far exceeds ordinary single function calls: the model needs to chain multiple tools within a single conversation (searching available components, retrieving component configuration schemas, validating parameter legality, assembling the final workflow), forming a multi-step calling chain. Any parameter error or incorrect calling sequence at any step could cause the generated workflow to fail or even trigger incorrect business logic.

Why Fine-Tune Instead of Directly Using Closed-Source LLMs?

Shopify's reasoning in the original article is very direct:

If you build AI products based on closed models, then anyone with an API key can achieve similar capabilities. True long-term differentiation isn't "we also connected a large model"—it's proprietary data, training approaches, infrastructure, and iteration speed.

This decision involves a classic trade-off in AI engineering: convenience vs. controllability. Directly calling closed-source model APIs like GPT-4 or Claude has the advantage of being ready out-of-the-box with no training infrastructure needed, but the disadvantages are equally clear—model behavior is uncontrollable (providers may update models at any time causing output drift), data privacy risks (business data must be sent to third parties), costs scale linearly with call volume, and no technical moat can be formed. Fine-tuning open-source models requires investment in GPU clusters, training pipelines, and evaluation systems, but once established, the enterprise gains a fully self-controlled model asset. Qwen-32B is a large language model open-sourced by Alibaba's Tongyi Qianwen team, with 32 billion parameters, performing exceptionally well on tasks like code generation and tool calling. Its Apache 2.0 license permits commercial use and fine-tuning, making it a popular base model choice for enterprise Agent fine-tuning.

The final results validated this judgment: the fine-tuned Qwen-32B model achieved 2.2x speed improvement, 60% cost reduction, and outperformed closed-source large models.

The Cold Start Dilemma: What to Do Without Conversation Data?

Training immediately hit a core problem: this capability hadn't been released yet—there were no real user conversations and no real interaction data. But to improve Tool Call accuracy, high-quality training samples were essential.

The Cold Start problem was first widely discussed in recommender systems—new users or new items have no historical interaction data, so the system can't make effective recommendations. In the AI Agent domain, the cold start challenge is even more severe: not only is user preference data missing, but even basic input-output sample pairs don't exist. Traditional supervised learning relies on large amounts of labeled data, while Reinforcement Learning from Human Feedback (RLHF) requires real human-machine interaction trajectories. When a brand-new Agent capability hasn't gone live yet, neither data source is available. Common industry strategies include: using strong models to generate synthetic data, transfer learning from similar tasks, or launching a basic version through rule-based systems first to collect data.

Shopify's approach was quite clever—rather than guessing what users would say, they first looked at what users had already built.

The answer was hidden within Shopify Flow itself. Many merchants had already manually created numerous Flows—some had been running, some were still in frequent use. These weren't conversation data, but they were the results of user intent. Users didn't leave behind questions, but they left behind completed answers.

It's like not hearing a customer place an order, but seeing the dishes on the table. The dishes aren't the menu, but they can help you reverse-engineer what the customer probably wanted to eat and how the chef prepared it.

Three-Step Method: Reverse-Engineering Training Data from Business Outcomes

Step 1: Filter Reliable Flow Samples

Shopify filtered high-quality Flows from the production environment with strict selection criteria:

• Run within the last 7 days, ensuring they're active business processes
• From merchants with multiple qualifying Flows, eliminating accidentally created low-quality samples
• Only one sample per similar description, ensuring coverage across different workflow types

Step 2: Use a Strong Model to Reverse-Generate User Requests

After obtaining a verified Flow, a stronger large model is used to reverse-generate a reasonable natural language request—essentially inferring "what might the merchant have said to get this Flow."

Using strong models to generate training data for weaker models is known in academia as a variant of Knowledge Distillation. Microsoft's Phi series of small models, Google's Gemma, and others have extensively used synthetic data for training. But Shopify's approach goes further: rather than simply having a strong model generate Q&A pairs, they perform reverse engineering starting from real business artifacts. The key advantage of this method is anchoring to reality—the generated training samples use actually running business processes as ground truth rather than being fabricated from nothing. In NLP, this technique of inferring questions from answers is also called Question Generation, which has been validated in reading comprehension, FAQ construction, and other scenarios. However, extending it to multi-step tool calling trajectory construction is a task with significantly higher engineering complexity, as it requires ensuring the logical reasonableness and parameter correctness of each intermediate call.

Step 3: Construct Complete Tool Calling Trajectories

This is the most critical step. It's not enough to have just the input (user request) and output (final Flow)—you also need to fill in the ideal Agent's complete tool calling process: what tools to search first, which candidates to review, which configurations to retrieve, then how to add conditions and actions, and finally how to assemble everything into an executable Flow.

The most important thing here isn't the final result—it's the intermediate path. With the path, the model knows how to search, filter, retrieve configuration structures, and assemble workflows when facing a user request.

What's Constructed Isn't Q&A Pairs, But Complete Calling Chains

The training data Shopify constructed isn't a short "question → answer" line—it's a complete chain:

Business outcome → User phrasing → Tool path → Final answer → Executable workflow

This synthetic dataset was ultimately used to fine-tune Qwen-32B. Additionally, Shopify built a benchmark test set containing 300 handcrafted examples covering the expected range of Flow usage, evaluating three categories of metrics:

1. Semantic correctness: Does the generated workflow semantically accomplish what the user intended?
2. Syntactic correctness: Are there format errors that would cause execution failures, such as malformed conditions, incorrect references, or invalid configurations?
3. Latency: How long does it take from the user making a request to the workflow being delivered?

The three evaluation metrics Shopify designed reflect core principles of enterprise AI system evaluation: you can't just look at "does the model think it's correct"—you also need to check "can the system actually run it" and "can the user tolerate the wait." Semantic correctness evaluates intent understanding—whether the generated workflow truly accomplishes what the user wanted, typically requiring human review or an LLM-as-Judge approach (using another large model as the evaluator). Syntactic correctness is a hard engineering constraint—a JSON format error, a reference to a non-existent field, or a type-mismatched parameter will cause the workflow to fail in Shopify's backend. The latency metric directly relates to user experience; in interactive Agent scenarios, response times exceeding 10 seconds significantly reduce user trust and willingness to use the system. While 300 handcrafted benchmark test samples may not seem large in number, it represents a relatively high investment in enterprise evaluation—each sample requires domain experts to manually write and verify, ensuring coverage of edge cases and common error patterns.

Implications for Enterprise AI Deployment

Shopify's cold start methodology has reference value for all teams looking to build enterprise-grade AI Agents:

If you don't have interaction data, don't just wait for users to speak up. Start from users' existing output results and reverse-engineer training samples.

Real enterprise data might be hidden in approval workflows, configuration tables, historical tickets, or already-running automation rules. You need to first recognize these as business artifacts, then translate them into training samples.

However, the original article also candidly reminds us: this is only the first step. After bootstrapping with this approach, significant gaps still need to be closed. Because reverse-engineered user requests are ultimately synthetic data—they can never fully represent how real users will speak once the feature goes live. Distribution shift in synthetic data is a widely studied problem—after training on synthetic data, models may experience performance degradation when facing real users' diverse expressions, ambiguous intents, or even incorrect descriptions. Therefore, the critical action after cold start is to establish a real data collection and feedback loop as quickly as possible, continuously iterating the model with online data.

The core engineering judgment can be summarized in one sentence: Much of enterprise AI training data isn't collected—it's mined and translated from business systems. This approach of "reverse-engineering intent from outcomes, then filling in paths from intent" is the most pragmatic cold start strategy before real user data accumulates.

Key Takeaways

• Shopify solved the Agent training cold start problem by reverse-engineering from existing merchant Flow business outcomes—when real conversation data is unavailable, they inferred intent and tool calling paths from users' completed outputs
• The fine-tuned proprietary model on Qwen-32B achieved 2.2x speed improvement and 60% cost reduction while outperforming closed-source large models, validating the long-term value of self-directed training
• Training data construction follows three steps: filtering reliable Flow samples from production, using a strong model to reverse-generate user requests, and completing full tool calling trajectories (intermediate paths matter more than final results)
• Enterprise AI training data often isn't directly collected—it's mined and translated from business systems like approval workflows, configuration tables, and historical tickets
• Synthetic data still has limitations and cannot fully represent real user behavior; cold start is only the first step, and subsequent efforts are still needed to close the gap with real-world scenarios