Stop Overpaying for AI Intelligence: A Complete Cost Optimization Guide from OpenAI DevDay

At the outdoor stage of OpenAI DevDay, Mandeep and Saptal from the AI deployment engineering team shared a topic that most developers overlook — how to genuinely optimize the cost of AI applications. Their core message was direct: most people fixate on the wrong metric from the start, and end up paying an unnecessarily high price for intelligence.

The Biggest Mistake: Watching "Cost per Token" Instead of "Cost per Task"

The most common mistake developers make is opening the pricing page, comparing input/output token unit prices, and picking the cheapest model. Mandeep pointed out that this approach misses a critical truth: a model that's cheaper per token isn't necessarily cheaper for completing an entire task.

There are two reasons for this: a cheaper model may consume or produce more tokens to complete a task, or it may simply fail to complete the task at all — ultimately requiring human intervention. In either case, the real cost far exceeds the raw token bill.

Mandeep used a relatable personal experience to illustrate the point: his flight was cancelled, and when he contacted the airline, he encountered a customer service chatbot. Despite patiently explaining that he wanted to rebook, the AI simply couldn't understand his request, and he eventually said, "I need to speak to a human." A task that AI was supposed to handle ended up carrying the additional time cost of a human agent. He concluded that the airline's real focus should be on cost per task — a number that includes the time and cost of human intervention.

He also cited two customer cases to support this logic. Perplexity reported in their research benchmarks that a newer model delivered 9% higher accuracy at half the cost compared to an older model. Notion, by continuously migrating to the latest model series (e.g., from 5.5 to 5.6), achieved higher accuracy and cut cost per task in half. In other words, a more expensive model can actually be more efficient when measured on a per-task basis.

How to Match the Right Model to a Task

Choosing the right model is the first major decision in cost optimization. Mandeep's methodology is: first clearly define the task, then define the accuracy level the business expects from it.

Returning to the airline example, the goal might be "have the chatbot resolve 80% of inquiries." Once a clear accuracy threshold is set, the next step is finding the model, harness, and configuration combination that meets that threshold at the lowest cost.

This is where he introduced an important concept — the Pareto curve. The axes represent accuracy and cost, which are in tension with each other. Developers need to test different models, settings, and frameworks, measure the per-task accuracy of each configuration, and then find the inflection point that achieves the minimum cost while meeting the accuracy threshold.

For tasks that are straightforward to evaluate — such as simple classification or extracting a few fields from a document — smaller models like Luna are often sufficient. For tasks requiring higher intelligence, you can start from the flagship model 6 Astra and work your way down. OpenAI emphasized that its model lineup — 6 Astra (frontier flagship), 6.1 Sol (high value), and Luna — all sit on the Pareto frontier of cost vs. accuracy.

Luna Maxing: Don't Underestimate Small Models

An interesting term that came up was "Luna maxing." Mandeep noted that many customers were surprised to find that Luna running at high or max reasoning effort could outperform many previous-generation or competing larger models. As a result, some customers use Luna as their default model for developer coding, only switching to a more powerful model when higher intelligence is truly needed. For 80–90% of workflows, Luna performs remarkably well.

The Pareto curve originates from the concept of Pareto efficiency in economics, and is used here to describe the trade-off boundary between accuracy and cost. Each point on the curve represents a model/configuration combination, meaning: the highest accuracy achievable at a given cost, or the lowest cost achievable at a given accuracy. Points on the curve are called "Pareto optimal," meaning you cannot improve one metric without sacrificing the other. Configurations that fall inside the curve (off the frontier) imply room for improvement — you can find alternatives with the same cost but higher accuracy, or the same accuracy but lower cost. In practice, developers need to draw this curve through systematic benchmarking: run the same set of representative tasks across different models, reasoning effort levels, and prompt structures; record the accuracy and cost of each configuration; and plot them to visually identify the lowest-cost inflection point that meets the business's accuracy threshold.

Four Cost Levers Beyond Model Selection

Saptal emphasized that costs can vary enormously within the same model family, depending on how much information you ask it to process and how much it outputs. He introduced four concrete levers.

1. Prompt Caching: Cache the repeated portions of your input prompt (such as a company policy document) so the model can reuse the pre-processed result of that shared information, while freshly processing each new piece of input and generating a new response. This is the highest-impact lever — OpenAI's pricing shows that cached inputs can save up to 90% of costs.

2. Programmatic Tool Calling: Saptal posed a question — when an agent calls multiple tools to process an order or look up inventory, does the model really need to personally read and compare every intermediate result? The answer is usually no. Programmatic tool calling lets the model run a small program to coordinate tool calls and organize results, returning only a packaged report to the model for judgment. This avoids stuffing every piece of intermediate data into the context and reduces the number of model calls. Legal AI company Clio reported a 38% reduction in prompt tokens on multi-step document analysis with no quality loss.

3. Reasoning Effort: Think of this as a dial you can tune. Note that reasoning effort has nothing to do with the length of the answer — even a brief answer can require significant reasoning. The recommendation: start with low reasoning effort for routine tasks, run evaluations to check whether quality holds; reserve the extra token cost for difficult or differentiated tasks. The core approach is making decisions through iterative evaluation runs.

4. Batch Processing / Flex API: The key question is "how long are you willing to wait?" If you need to process thousands of documents without requiring immediate results, OpenAI's batch processing API offers a 24-hour processing window at 50% lower token prices than standard synchronous requests.

Saptal's advice: when choosing a model family, also ask — "How much work am I making the model do?" Walk through a complete task in your application, examine inputs and outputs, identify redundant workflows and tool call counts, then change one setting at a time, iterate, and run evaluations. The goal isn't simply to reduce tokens — it's to get successful results at lower cost within acceptable latency.

The core idea behind Programmatic Tool Calling comes from separating the "orchestration layer" from the "reasoning layer." In traditional agentic workflows, the result of every tool call is appended verbatim to the context, which the large model then reads to decide the next action — causing context to grow linearly with each tool call, driving token costs up sharply. Programmatic tool calling inserts a lightweight piece of code logic between the model and the tools: the program handles scheduling tool calls, aggregating intermediate results, and filtering out irrelevant information, delivering only a condensed summary or conclusion back to the model. This is consistent with the software engineering principle of "separation of concerns" — let precise-executing code handle deterministic logic, and let expensive model reasoning focus only on the decision points that truly require semantic understanding. This pattern is especially effective for multi-step data queries, document analysis pipelines, and similar scenarios.

A Deeper Look at Prompt Caching: Common Developer Mistakes

Given the outsized impact of prompt caching, both speakers dedicated extended time to it. Audience questions revealed that nearly all developers were using prompt caching — it's on by default — but many weren't getting the most out of it.

The primary principle is to keep the beginning of your prompt as consistent as possible, so the cache can reuse the pre-processed shared instructions while new information is reprocessed. The case of customer Blitzy (an AI software development company) is compelling: by switching from single structured output calls to a Luna tool-calling loop, they raised their cache hit rate from 24% to 90%, achieving 8.5x fewer output tokens across thousands of production calls, 87% lower cost compared to GPT 5.4 mini, and 2.2x more context processed simultaneously.

What to Do When Prompts Change Frequently

For earlier model families, automatic caching was on by default with the system choosing breakpoints at token intervals — developers had no control over boundaries. With the GPT 5.6 series, explicit breakpoints were introduced — you can manually set cache points after stable content and before frequently changing task inputs, so stable parts are reused while changing parts are reprocessed.

For even finer control, there's also explicit-only mode, which lets you select exactly which parts to cache, avoiding caching content that changes frequently and won't be reused later.

With GPT 6 Astra, all previous capabilities are retained, plus one key addition: you can change reasoning effort mid-conversation without breaking the cache. Additionally, developers have long been able to change available tools via parameters like allowed_tools without breaking the cache.

OpenAI has also launched a prompt caching dashboard to measure and track cache hit rates for your application, along with a diagnostic API that pinpoints where cache misses occur between two different queries, helping you optimize more precisely.

Prompt Caching works as follows: when a model receives an input, it tokenizes and pre-processes it, generating a set of intermediate representations — commonly called a KV Cache (Key-Value Cache). If a subsequent request's input prefix is identical to a previous one, the model can directly reuse the already-computed KV Cache, skipping recomputation for that portion and dramatically reducing computation and associated costs. This also explains why "keeping the beginning of your prompt consistent" is so critical — caching is prefix-matched. Fixed content like system prompts and company policy documents should go first, with changing user inputs at the end, to maximize hit rates. A cache miss means that portion of tokens is billed at the full price. Prompt caching delivers the greatest benefit for long system prompts (such as embedding an entire knowledge base or operations manual), since that fixed content typically accounts for the vast majority of input tokens.

An Actionable Checklist

The closing recommendations from both speakers were clear and executable:

• Start by defining the task you want the model to complete, and establish the accuracy threshold the business expects;
• Build a representative set of tasks that reflects real usage, and experiment with different models, settings, reasoning effort levels, and prompts;
• Plot the Pareto frontier and find the model, configuration, and framework combination that achieves minimum cost at your accuracy threshold;
• Test one improvement at a time — lock in what works and iterate continuously, re-running evaluations as your application and the models evolve.

At the end of the day, the core message of this talk remained consistent throughout: don't overpay for intelligence. The metric that truly matters is reliably completing tasks at lower cost within sustainable latency.