OpenAI API Development Methodology: Diagnose First, Then Choose Your Tech

At the first session of OpenAI's official API Bootcamp series, Ellie from the Enablement team made a counterintuitive but valuable point: building API applications should work like debugging any engineering problem — understand the real problem first, then consider whether you need more infrastructure. Rather than piling on technical buzzwords, the talk walked through the complete thought process from fuzzy idea to production-ready application, using a fictional customer service team called Acme as a running example.

Three Common Traps: Don't Rush to Agents and RAG

Many teams open with "we need an Agent" or "we should add RAG," yet haven't even defined what the first useful version should do. Ellie used a sharp analogy: it's like deciding to renovate the entire kitchen before figuring out why your coffee tastes bad — maybe you do need the renovation, but maybe you just need to clean the coffee machine.

She identified three typical traps: vague use cases, infatuation with a particular technology, and no clear way to validate results. None of these can be solved by piling on more infrastructure. This is the foundational stance the talk returned to repeatedly: technology choices should be driven by requirements, not by how "advanced" a technology is.

The BCA Diagnostic Framework: Behavior, Context, Action

The heart of the talk is a diagnostic mental model that breaks problems into three distinct "lanes," with a quality dimension running through all of them.

Behavior: The model has the information it needs, but the response doesn't follow the right instructions, format, or decision criteria. Like asking a colleague for a one-sentence status update and getting a five-page document — they don't lack the information, they lack a clear task brief.
Context: The model is missing information it needs to complete the task. Like asking someone to explain a return policy without giving them the actual policy document.
Action: A correct answer isn't enough — something needs to happen beyond the response, like querying an order or opening a support ticket.
Quality: The question that runs through all three — how do we know it's actually working?

Ellie was careful to stress that these are not maturity levels, but different diagnostic directions. You don't need to "graduate" from Behavior to Agent — you choose the right fix based on where the specific failure is.

From Vague Goal to a Testable First Slice

Using Acme's customer service team as an example: "improve customer service with AI" is a reasonable vision, but it doesn't tell developers where to start. Ellie proposed six scoping questions to work through before picking any model or architecture: who's involved, what does the current state look like, what signal would indicate improvement, what's already been tried, what was learned, and what steps must stay in human hands.

With this framing, an overly broad goal ("improve customer service operations with AI") can narrow into an achievable slice: classify incoming tickets, flag policy-sensitive cases, and preserve human review at key decision points. The corresponding success signal: faster manual triage without a drop in routing accuracy. It's like sorting your inbox faster without accidentally burying important emails.

Four Control Knobs at the Behavior Layer

When the first requirement is making consistent, reliable routing decisions from information already in the ticket, the problem sits at the behavior layer. Ellie introduced four practical controls: Prompting, structured outputs, model selection, and reasoning intensity. She was clear these are different knobs — not a checklist where you crank everything to maximum.

A good prompt is less like a secret incantation and more like a well-written engineering ticket: clearly state the goal, what information the system has available, where the boundaries are, and what a completed result looks like.

A good prompt should define goals and boundaries as clearly as an engineering ticket

But clear instructions alone can't conjure missing policy information, nor guarantee a stable response format. That's where structured outputs come in — giving the response a "contract": which fields, what types, which values are allowed. The value is that your application doesn't have to guess whether the model named the field category, ticket_type, or issue. Ellie demonstrated using the Responses API with text_format to define a schema, allowing outputs to feed directly into queues, dashboards, or downstream processes.

Her stance on model selection was refreshingly pragmatic: don't always pick the one with the flashiest name. Treat it like any other dependency — run a representative set of tickets through different candidate models and compare classification accuracy, sensitive case recall, latency, and cost. "Evaluations, not model labels, make the decision." The same logic applies to reasoning intensity — change one setting, rerun the same cases, and let the evidence speak.

The Context Layer: What RAG Actually Is

When the problem is that the model lacks information, you enter the context layer. Ellie cleared up a common misconception: context doesn't automatically mean you need to build a retrieval system. If you're just summarizing one article, just give it the article. It's only a different matter when you need to find the right passage across tens of thousands of constantly changing documents.

If the information fits on a sticky note, don't build a search engine for it

For RAG (Retrieval-Augmented Generation), her explanation was simple and accurate: find relevant information, provide it with the request, generate an answer based on it. Like a friend asking what time the package locker closes — you don't memorize the entire resident handbook, you look up the relevant page and answer.

She recommended starting every context decision with one question: where does the missing information actually live? Short, known policy text can be inlined directly. Public, time-sensitive information can use web search. For information stored in approved company documents, hosted file_search is a pragmatic starting point. And only when you need strict control over ranking, permissions, or integration with existing data systems does it make sense to build your own retrieval stack. She compared it to food delivery: sometimes ordering in is enough, sometimes you want to pick your own ingredients — only when you truly need that level of control should you run the whole kitchen.

The key evaluation principle: retrieval quality and answer quality should be assessed separately — finding the right book and reading it correctly are two different things.

RAG (Retrieval-Augmented Generation) is designed to address two inherent limitations of large language models: the knowledge cutoff date of training data, and the inability to directly access private or real-time information. The workflow typically has two phases: an offline phase that chunks documents and converts them into vector embeddings stored in a vector database; and an online phase that converts the user's query into a vector, retrieves the most similar chunks, appends them to the prompt, and has the model generate an answer.

This mechanism looks elegant, but failures in practice often concentrate at the retrieval step: chunks that are too coarse, embedding models that don't match query semantics, or relevant documents that can't be indexed due to permission boundaries — all of these cause the model to "find the wrong page." This is exactly why Ellie emphasizes assessing retrieval quality and answer quality separately. A fluent-looking generated answer may be built on completely wrong retrieval results, and these issues only surface when you specifically test the retrieval layer.

The Action Layer: Who Controls What the System Is Allowed to Do

Actions are only introduced when a useful result requires something to happen beyond generating a response. Ellie stressed that actions are not "the next level everyone graduates to." Her primary concern was: who controls what the system is allowed to do?

She distinguished between several capability carriers: Code Interpreter generates charts in a sandbox but can't update Acme's live systems; custom functions handle order lookups and opening review tickets; MCP servers provide interfaces to connected external services. A subtle but important point — the existence of a tool doesn't automatically make a use case "action-oriented." file_search serves a context need, while "create a refund review ticket" is an action, because it changes something outside the response.

If the model read the policy correctly but returned the wrong format, that's still a behavior problem

In the three-step function-calling flow, the boundary is critical: Acme defines the capability in code, the model requests a call with structured parameters, and Acme's application decides whether to execute it. Like submitting an expense report — submitting the claim isn't the same as approving payment; finance still checks against policy. She intentionally designed a function that can only open a review ticket, with no authority to approve a refund — like being able to submit a purchase request but not being handed the company credit card.

On Agents, she offered a clear criterion: Agents become useful when the next approved step depends on what the system has discovered along the way. A chatbot handles "what's the travel policy?"; a workflow handles fixed-step cancellation and refunds; only adaptive paths like "compare rebooking options" warrant a controlled Agent loop. She also demonstrated using the Agents SDK to configure input guardrails, restrict the tool list, and set max_turns to prevent infinite loops — like giving a temporary contractor a time-limited badge with restricted access.

MCP (Model Context Protocol) is an open protocol proposed by Anthropic in late 2024, designed to give large language models a standardized way to connect to external tools and data sources. The design philosophy is similar to the USB specification: a tool provider implements an MCP server once, and various models or Agent frameworks can discover and call these capabilities through a unified interface, without building a custom integration layer for each.

In practice, the key difference between MCP and custom function calling is the granularity of management: function calling typically requires developers to explicitly define each tool's schema and execution logic in code, while an MCP server can dynamically expose a list of tools — better suited for connecting to existing external services like GitHub, Slack, or databases. In the talk, Ellie listed it as the option for "providing an interface to connected external services," with the core consideration still being permission boundaries — the capability scope of an MCP server needs to be defined at the architecture level in advance, rather than being discovered and expanded by the model at runtime.

Quality and Evaluation: Adding "Regression Tests" to AI Behavior

Quality isn't a stage bolted on at the end — it's a habit that runs throughout the process, and evals are the reproducible mechanism for it. A complete eval has four parts: dataset, system under test, scorer, and metrics. The dataset should include both typical requests and the edge cases you actually care about — one late order outside the return window is worth more than 20 nearly identical simple examples.

Ellie repeatedly emphasized one discipline: change only one variable at a time, and rerun the same set of cases. It's like investigating why a cake keeps falling — if you change the oven temperature, the recipe, and the pan at the same time, you'll get different results but won't know which change mattered. The scoring method should also match what you changed: for prompts, check whether classification is correct; for structured outputs, validate both the schema and field values; for retrieval, check whether the right source was found and whether the answer is faithful to it; for tools and Agents, verify the tool selected, the parameters, the stopping condition, and any side effects.

After changing a model or prompt, rerun the same cases to verify that key behaviors haven't regressed

She also cited the real-world case of BlueJ: in the tax research domain, answers must come from trusted, current sources, making context and quality the central concerns. Results included roughly 2.7 hours saved per user per week, coverage across 3,000 firms in three countries, and an error rate below one in seven hundred. The point isn't to copy the architecture — it's to identify the real gap, choose the right implementation, and keep testing whether it works.

LLM-as-a-judge is a widely adopted automated evaluation approach: an independent model (typically a more capable version) scores the outputs of the system under test or renders pass/fail judgments. Its advantage is handling open-ended outputs — for questions like "is the classification correct," rule matching is sufficient, but for "is this answer faithful to the source document," manual review of every case is prohibitively expensive, while an LLM scorer can cover large batches quickly.

Several common biases to watch for when using this approach: the scoring model may prefer longer, more confidently worded answers over more accurate ones; when the system under test and the scoring model come from the same provider, there may be systematic stylistic preferences. For this reason, building an LLM-as-a-judge pipeline typically requires first calibrating the scorer's reliability against a set of human-annotated samples before deploying it as part of an automated pipeline.

Getting to Production: A Practical Development Habit

The entire talk ultimately converged on a four-step developer habit: Diagnose (what does the first useful version have to get right), Choose (is the first step behavior, context, action, or further scoping), Defer (which complexity can wait), and Verify (which real case will you use to test it).

In the Q&A, Ellie added several practical notes: RAG reliability depends on whether sources are trustworthy, how frequently they're updated, and whether you flag cases for human review when evidence is insufficient. Human-in-the-loop should focus on high-stakes, irreversible decision points, with a clear definition of "success" established upfront. The most common mistake going from prototype to production is stuffing in too much irrelevant context, and ignoring the tradeoffs between latency, cost, and model selection at scale. On LLM-as-a-judge versus human-in-the-loop, she doesn't think production environments require both — low-risk actions can be automated with guardrails and monitoring, while high-risk decisions warrant human oversight.

The value of this talk isn't any specific API call — it's building a kind of judgment: knowing what you're building, why a given technology fits, what can wait, and how to tell whether it's actually working.