Breaking Down RAG Service Costs: Token, Retrieval, and Observability Pricing Explained

When building a RAG (Retrieval-Augmented Generation) service, most people focus on model selection, but what really determines the bill is often the parts that are easy to overlook. A detailed cost model breaks down every expense from tokens to vector databases to LLM observability tools—and the results may upend your intuition.

According to the analysis, a typical RAG query—2,000-token prompt, 2,500-token retrieved context, 400-token output—costs about $0.013 on a mid-tier model. Add retrieval, reranking, and observability, and at production scale you're looking at roughly $13 per 1,000 queries.

LLM Tokens: The Hidden Cost That Eats Most of the Budget

The cost model breaks down monthly RAG service costs into three major categories: per-query usage costs (LLM + query embeddings + vector reads + reranking + observability), fixed platform fees (plan minimums, seats, storage), and corpus maintenance plus human labor.

The per-query token budget consists of: a 1,500-token system prompt (cacheable for a discount), roughly 500 tokens of user messages and conversation history, a 2,500-token retrieved context (five 500-token chunks), and a 400-token output—totaling 4,900 tokens per request. The retrieved context is the largest source of input, and every extra chunk you stuff in gets paid for on every single query.

Priced on a mid-tier model (like Claude Sonnet 5.5, gpt-6.1-sol): $2 per million input tokens, $10 per million output tokens, $0.1 per million cached input tokens. With prompt caching enabled, a single query costs about $0.01033; without it, $0.0130—caching saves roughly 20%.

The price spread across tiers is the biggest variable in the budget. The same query costs just $0.00052 on a small model, $0.01033 on a mid-tier model, $0.02066 on Claude Opus 5.5, and a whopping $0.05236 on gpt-6-astra. Nothing about the query itself changed—only the model did. That 100x spread between small and large tiers is the single most consequential decision in your RAG budget.

Prompt Caching is a cost-optimization mechanism offered by major LLM providers: when the prefix of a request (usually the system prompt, fixed instructions, or a long document) matches a previous request exactly, the server reuses the cached key-value (KV) state instead of recomputing attention for those tokens, billing them at a price far below standard input—the cached input price cited above ($0.1 per million tokens) is just one-twentieth of standard input ($2 per million tokens).

But caching comes with conditions: the prefix must match byte-for-byte, and caches typically have a time-to-live (e.g., 5 minutes to 1 hour) after which they must be rewritten. This is exactly why caching delivers less benefit in RAG scenarios than in long-prompt chatbots—the retrieved context changes on every query, and placing it early in the prompt would break cache hits for everything after it. So in practice, engineers put the static system prompt first to benefit from caching and place the dynamically retrieved document chunks afterward.

Retrieval Costs: Vector Search Is Cheap, Reranking Is the Surprise

Embedding costs are nearly negligible. OpenAI text-embedding-3-small is priced at $0.02 per million tokens; a 500-million-token initial corpus costs $10 one-time, monthly refreshes cost just $1, and per-query embedding cost is $0.000001. What's genuinely expensive isn't embedding itself, but the reindexing engineering time when you adjust chunking strategy or switch models.

Taking Pinecone serverless as an example, 1 million vectors (1536 dimensions) make up about a 7.14 GB index, each query consumes about 7.14 read units, per-query read cost is about $0.000114, storage is about $2.36/month, and the standard plan has a $50/month minimum. At 10,000 and 100,000 queries/month, you're paying that $50 minimum either way; even at 1 million queries, the vector database only costs about $120/month.

Reranking is the most commonly forgotten expense line. A cross-encoder reranker selects 5 from 50 candidates; Cohere Rerank 4 Fast is priced at $2 per 1,000 searches, or $0.002 per query. At 100,000 queries/month that's $200, and at 1 million it soars to $2,000—roughly 15% of usage cost, an order of magnitude more than the vector database.

Reranking is the second-stage fine-ranking step in a RAG retrieval pipeline. To avoid missing relevant documents, the first-stage vector retrieval (also called recall) typically uses a Bi-Encoder to quickly pull roughly 50 candidates from millions of vectors—it encodes the query and documents independently into vectors and computes cosine similarity. It's fast but limited in precision, because there's no interaction between query and document.

Reranking uses a Cross-Encoder instead: it concatenates the query with each candidate document and feeds them into the model together, letting the two fully interact in the attention layers, producing a more precise relevance score, then selecting 5 out of 50 to send to the LLM. Precision improves significantly, but the trade-off is that you can't pre-build an index—each candidate requires a full forward pass—so it can only be used on small candidate sets and is billed per "search." That's why its unit price looks modest but it becomes the third-largest line item once you scale.

Observability: Per-Unit Billing vs. Per-Trace Billing

Without tracking retrieval quality, latency, and token consumption, a RAG service can't run in production. The two leading managed solutions have fundamentally different pricing logic.

Langfuse Cloud bills per unit—every trace, observation, and score counts as one unit. The Core plan is $29/month for 100,000 units, with overages at $8 per 100,000 units. LangSmith bills per seat and per trace. The Plus plan is $39/seat/month, includes 10,000 base traces, with overages at about $2.50 per 1,000 traces.

The real-world difference is substantial: at 10,000 queries/month, assuming 8 units per query, Langfuse costs just the $29 plan fee, while LangSmith costs $117 for three seats. At 1 million queries, Langfuse is about $661 and LangSmith about $2,592. The gap keeps widening as traffic grows.

Observability can quietly eat 5% to 20% of usage cost. A common control is to sample 10% to 20% of production traffic for tracing while logging token data at 100%.

LLM Observability is a monitoring and tracing system specifically for LLM applications. Traditional APM only sees the duration and status code of an HTTP request, whereas a single user query in RAG internally unfolds into a call tree: query rewriting → embedding → vector retrieval → reranking → prompt assembly → LLM generation, and each step needs to record inputs, outputs, latency, and token consumption.

In the industry, a Trace represents one complete end-to-end request, a Span / Observation represents a single step in the tree, and a Score records a quality rating from a human or a model. The phrase "8 units per query" above refers to this tree having roughly 8 billed nodes. Differences in vendors' metering definitions are exactly the source of cost gaps: Langfuse bills for every unit, whereas LangSmith charges a flat per-seat fee and only bills base traces beyond the included quota—so per-seat billing is more expensive at low traffic, and per-unit billing is more expensive at high traffic.

Real Bills for Three Scale Scenarios

Based on a mid-tier model (with prompt caching enabled) and Langfuse observability, the line-item breakdown for three scenarios is as follows:

10,000 queries/month: LLM tokens $103.30, reranking $20, observability $29, vector database $50 (plan minimum), query embeddings $0.01, corpus refresh $1, total $203.31. The vector database accounts for a quarter of the bill—purely because of the minimum. Add a 15% buffer and it's $233.81.

100,000 queries/month: LLM tokens $1,033, reranking $200, observability $85, vector database still $50, total $1,369.10, or $13.69 per 1,000 queries.

1 million queries/month: LLM tokens $10,330, reranking $2,000, observability $661, vector database finally exceeds the minimum at $119.52, total $13,112.52. LLM tokens account for nearly four-fifths, reranking takes most of the remainder, and embeddings and storage are almost negligible.

Model choice has an equally dramatic effect on the bill. At 1 million queries/month, a small model totals about $3,300, a mid-tier model about $13,100, Claude Opus 5.5 about $23,400, and gpt-6-astra about $55,100—roughly a 17x difference between the cheapest and most expensive tiers. Cost per 1,000 queries declines with scale simply because fixed fees get amortized; beyond about 100,000 queries, the bill becomes almost entirely variable cost.

What Costs More Than Tokens at Low Traffic: People

A production-grade RAG service requires ongoing investment: evaluation set maintenance, chunking and prompt tuning, regression testing when models update, cost monitoring, and incident response. At 0.5 full-time AI engineers, roughly 20 hours per week, labor costs vary hugely by region.

In the US, at a fully loaded cost of about $150/hour, that's about $11,250/month; Central and Eastern Europe outsourcing at about $80.50/hour comes to about $6,440; Latin America at about $77.63/hour is about $6,210; Asia at about $41.40/hour is about $3,312.

At 10,000 queries/month, usage costs are about $200 while maintenance labor runs between $3,300 and $11,250—the real question you need to answer is "who maintains this system," not "which model do we pick." By 1 million queries/month, usage costs (about $13,000) exceed labor costs in every region, and only then do model routing and prompt optimization start to justify an engineer's time.

Optimization Priority: Tokens First, Then Reranking, Everything Else Later

To see which decisions matter most, adjust line items one by one in the 100,000-query mid-tier model scenario (baseline cost about $1,369/month):

Switching to a small model has the biggest effect, saving about $980/month (72%), because tokens dominate cost. Retrieving 3 chunks instead of 5 saves 1,000 input tokens per query, saving $200 (15%). Shortening answers from 400 to 250 tokens saves $150 (11%), because output tokens cost 5x input. Dropping the reranker also saves $200 (15%), but is very likely to hurt answer quality.

Some changes actually increase cost. Disabling prompt caching adds about $270 (20%). Switching to LangSmith with three seats adds about $257. Partitioning the vector index saves almost nothing at the 100,000-query scale, because you're still paying Pinecone's $50 minimum.

The core conclusion is clear: optimize tokens first, reranking second, and everything else can wait.

Model Routing is the engineering realization of the optimization priorities above: instead of sending every query to the same model, first use a lightweight classifier or heuristic to judge query difficulty—simple factual Q&A goes to a small model, and only complex reasoning or multi-hop questions escalate to mid- or large-tier models. Since the article notes that model tier selection is the biggest single variable in the budget (roughly a 17x spread from small models to the most expensive tier), even if only 30%–50% of traffic can be safely downgraded, that yields a double-digit overall cost reduction.

The hard part of routing isn't the technical implementation but quality assurance: you must rely on an evaluation set to continuously verify that downgraded queries haven't regressed in quality, and rerun regression tests when models update. This also explains why the article recommends prioritizing investment in evaluation sets and chunking tuning at low traffic rather than doing routing too early—without a reliable evaluation baseline, the savings from routing can't be trusted.

Frequently Asked Questions

How much does a RAG query actually cost? On a mid-tier model at 100,000+ queries/month, about $0.013 per query (roughly $13 per 1,000). Small models can drop below $0.003, while frontier models may exceed $0.05.

What's the biggest cost in a RAG system? At scale, LLM tokens—especially the input tokens from retrieved context. At low traffic, the engineering time to maintain and evaluate the system exceeds all usage costs combined.

How much does the vector database cost? For a roughly 7 GB index (1 million chunks), about $50 to $120/month across the 100,000 to 1 million query range. At low traffic, the plan minimum is the dominant factor.

Can prompt caching reduce RAG costs? Yes, but less dramatically than for chatbots with long static prompts, because the retrieved context changes on every query. In the model above, caching cut LLM costs by about 20%.

Should you self-host the vector database and observability stack? Self-hosting eliminates subscription fees but adds infrastructure and engineering time. At this scale, subscription costs are trivial relative to half an engineer's salary; self-hosting is usually better suited to data residency or compliance needs than to cost considerations.

One-line Summary

RAG service bills are dominated by tokens, and at low traffic, labor costs far exceed API fees.

Paragraph Summary

A detailed cost model breaks down the real expenses of a RAG service. A typical query (4,900 tokens, mid-tier model) costs about $0.013, and 100,000 queries/month costs about $1,369. Of the three major cost categories, LLM tokens account for 75%–80% of usage cost at scale, with retrieved context being the largest source of input tokens, while vector search itself is under 1%; reranking is a surprising third-largest item at about 15%, an order of magnitude more than the vector database. Model tier selection is the biggest single variable, with a roughly 17x difference between the cheapest and most expensive models. At low traffic the picture flips: at 10,000 queries/month, usage costs about $200 while maintenance labor runs $3,300–$11,250. The optimization priority should be tokens first, reranking second, everything else later, and prompt caching can save about 20% of LLM costs.