Stop Guessing with Your Agent: Building Measurable LLM Apps with DSPy + MLflow

When Agent "Improvements" Are Just a Feeling

Many engineers maintaining AI Agents in production have run into an awkward situation: they tweak a prompt, the model seems to perform better, but they can't explain exactly what improved or by how much. The next change might inexplicably make things worse. This "prompt-and-pray" development approach fundamentally builds system reliability on intuition and luck.

A three-hour hands-on workshop led by AI engineers Serj Smorodinsky and Brett Kennedy addresses exactly this blind spot. Both are co-authors of a book on LLM applications, and the workshop focuses on how to build Agents and LLM applications using structured methods — rather than endlessly tweaking prompts by trial and error.

Reddit original post screenshot

From Handwritten Prompts to DSPy Structured Programming

The workshop's core idea is to replace handcrafted prompts with DSPy signatures and modules. DSPy abstracts "what I want the model to do" into composable, optimizable program units — rather than fragile natural language text.

The significance of this shift: once a task is structurally defined, the specific wording of prompts is no longer something you need to manually debug over and over. Instead, it becomes something the framework can automatically optimize based on evaluation results. The workshop covers optimization at both the few-shot and instruction levels, giving model behavior improvements a solid, evidence-based foundation.

Establish a Measurable Baseline First

The first step in structured development isn't optimization — it's establishing a measurable baseline classifier. Without a baseline, any "improvement" has nothing to compare against. The workshop walks participants through building a baseline from scratch and making quantitative comparisons on top of it — this is the starting point for solving the "going by feel" problem.

DSPy (Declarative Self-improving Python) is an open-source framework developed by Stanford's NLP team. Its core philosophy is to abstract LLM calls into programmable modules rather than relying on handwritten prompts. In DSPy, a Signature defines the input and output types of a task (similar to a function signature) — for example, "take a piece of text as input, output a sentiment classification label." A Module is the basic unit that executes that task, analogous to a layer in a neural network. Multiple Modules can be chained together to form more complex pipelines.

DSPy's most critical capability is automatic optimization: it includes several built-in optimizers (such as BootstrapFewShot and MIPRO) that can automatically search for better few-shot examples or instruction phrasings based on your evaluation metrics — replacing the manual process of iteratively tuning prompts by hand. This means prompts shift from being "handcrafted artifacts" to "parameters that can be algorithmically optimized," with the entire optimization process driven by a clear objective function rather than an engineer's subjective judgment.

In LLM application development, a "baseline" refers specifically to the initial performance level of a system before any optimization, typically measured using a fixed evaluation dataset and a set of task-relevant metrics. A common pitfall is modifying prompts based on intuition without having a numerical anchor to determine whether a change actually worked — leaving all "improvements" in an unfalsifiable state.

The standard process for establishing a baseline includes: selecting or annotating a representative set of test samples, defining evaluation metrics aligned with business goals (such as accuracy, F1, or a custom LLM-as-judge score), and then running the current system to get a benchmark score. This score then becomes the reference point for all subsequent optimization — any prompt modification or model swap must be compared against the same dataset to determine whether it represents a real improvement or random variance.

Use an Evaluation Dataset to Clearly See Failure Modes

A baseline alone isn't enough — you also need to understand the conditions under which your system fails. The workshop includes a dedicated segment on building evaluation datasets with task-specific metrics, with an emphasis on how to directly identify failure modes from evaluation outputs.

This is a capability many teams overlook. Rather than vaguely concluding that "the model isn't great," an evaluation dataset lets you pinpoint specific error categories — is it struggling with long texts, or breaking down on certain edge-case inputs? Visualizing failure modes gives your improvement efforts a clear direction.

MLflow: Making Experiments Traceable and Reusable

Evaluation alone isn't sufficient — the experimentation process itself needs to be recorded. The workshop introduces MLflow for experiment tracking and trace management, ensuring every adjustment leaves a traceable record.

Combined with another practical capability of DSPy — saving and reusing optimized DSPy programs — teams can turn "well-tuned results" into durable assets rather than starting from scratch each time. This means the outcomes of optimization become reproducible and amenable to proper engineering management.

MLflow is an open-source MLOps platform led by Databricks, originally designed for traditional machine learning experiment management and recently extended with native support for LLMs and Agents. In LLM application scenarios, MLflow's Tracking module can record each experiment's parameters (such as model version used, optimizer configuration), evaluation metric scores, and complete input/output logs. The Tracing feature can capture the full call chain of every Agent step — including which tools were called and what the raw LLM inputs and outputs were — making it easy to retrospectively diagnose anomalous behavior.

The practical value of combining MLflow with DSPy lies in this: DSPy is responsible for generating optimized programs (containing the best few-shot examples and instructions), while MLflow persistently stores the program version, corresponding evaluation scores, and experiment context together. This allows team members to reproduce any historical experiment, and to make quantitative comparisons between different optimization strategies — rather than relying on memory of "that one time it seemed better" to make decisions.

Explaining Reliability to People Who Don't Care About Prompt History

One particularly interesting item on the workshop agenda is: how to explain LLM/Agent reliability to stakeholders who simply don't care about your prompt revision history.

This touches on a very real pain point. Business stakeholders or managers won't care which word you changed — they just want to know whether the system is reliable and what quantifiable gains a given change delivered. When you have a baseline, evaluation metrics, and experiment records in hand, that conversation can be grounded in data rather than talking points.

Who Should Attend

The organizers are clear about the target audience: if you're maintaining Agents in production but can't explain why a given change helped or hurt, this workshop is designed to fill exactly that gap. The three-hour hands-on format means it prioritizes practical application over pure theory.

For engineers who are already building applications with LLMs and are starting to be challenged by maintainability and reliability concerns, the DSPy + MLflow combination represents a more engineering-driven approach: pulling Agent development from "art" back into "measurable engineering." Of course, the complete workshop agenda and details should be referenced from the original post's comment section — this article is based solely on publicly available summary information.