Prompting, RAG, or Fine-Tuning? Diagnose the Failure First

When a large model application produces a wrong answer, many teams' first instinct is to "switch to a bigger model" or "start fine-tuning." But a single wrong answer is just one observation, not a root cause. Before you start rewriting prompts, wiring in retrieval, or kicking off training, figure out where the failure actually comes from—missing context, ambiguous labels, insufficient base model capability, or something that isn't the model at all.

Don't Mistake Symptoms for Causes

The same kind of error can have completely different root causes. Unclear instructions and retrieval that missed the key documents both show up as "answering the wrong question." Missing live values and missing private knowledge both show up as "making things up." A base model that isn't capable enough and training data with noisy labels can also produce seemingly similar mistakes.

The source material provides a diagnostic comparison table with a clear logic: look at the evidence first, then choose the intervention.

Observed Problem	Evidence to Check First	Candidate Intervention
Unclear instructions or wrong style	Prompt wording, examples, instruction conflicts, task tests	Improve prompts and evaluate the change
Answer requires private or changing documents	Source versions, access filters, retrieval coverage, citation support	Retrieve relevant documents via RAG and use as context
Answer requires live values	Authoritative service, timestamps, units, response structure	Call a tool or API at request time
A specific rule must be strictly enforced	Approved rule specification, inputs, edge cases, authorization	Implement in deterministic code and test
After baseline optimization, stable task patterns remain weak	Label quality, representative samples, model capability, holdout results	Evaluate fine-tuning against the baseline

A common misconception: if an error occurs frequently, it must come from pretraining. The source material explicitly warns to check labels, missing context, preprocessing, and task definition first—then consider swapping the base model or adding training.

What Each Approach Solves

Prompting provides instructions and examples at inference time. It's the lowest-cost and fastest to iterate, well-suited for expression style, format constraints, and task boundary descriptions. It doesn't change model weights, so it can't introduce private knowledge the model doesn't have, nor can it guarantee real-time accuracy.

RAG retrieves relevant content and injects it into the model input. It's suitable for private documents or frequently changing materials. Its evaluation focus isn't the model itself, but source versions, access filters, retrieval coverage, ranking quality, and whether citations actually support each factual claim. The source material particularly emphasizes: retrieval doesn't equal correct answers, let alone compliance—source coverage and human review remain necessary.

Fine-tuning uses training samples to change the model's learned behavior. It's suitable for cases where "stable task patterns remain weak after baseline optimization." It's not appropriate for refreshing changing facts, nor for strictly enforcing precise rules. The source material mentions LoRA: it freezes base weights and learns low-rank updates—but freezing the base doesn't mean behavior stays unchanged. Once updates are applied, gains on the target task and regressions on other tasks must both be measured empirically.

The three can be combined: a fine-tuned model can still use prompts, retrieval, and tool calls.

RAG (Retrieval-Augmented Generation) works by first retrieving document fragments relevant to the question from an external knowledge base, then concatenating those fragments into the model's input context, letting the model "answer while looking at the material." It decouples knowledge storage from language generation: model weights stay unchanged, and knowledge updates only require updating the index. This makes RAG particularly suitable for private documents, frequently changing policy terms, internal manuals, and similar scenarios. But retrieval quality determines the answer ceiling—if the retrieval stage fails to recall key documents, or recalls outdated versions, even the strongest model can only answer based on wrong material, and may even use fluent language to package incorrect information convincingly.

LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning method. It doesn't update all the base model's weights; instead, it inserts small low-rank matrices alongside certain layers and trains only those small matrices. This dramatically reduces GPU memory and storage costs, and allows the same base to host multiple task adapters. It's worth emphasizing that LoRA freezes the base weights, but once the adapter takes effect, the model's output distribution on the target task changes—which may simultaneously affect performance on other tasks, making regression testing essential.

Decision Flow: Start from Evidence

The decision flow from the source material can be summarized as: first check failure evidence, then separately evaluate prompt optimization, retrieving relevant documents, calling authoritative live values, enforcing rules with code, and evaluating fine-tuning for stable patterns—finally comparing results and operational costs holistically.

The key point: one symptom may point to multiple candidate interventions, and each change must be validated with task-level evaluation. You can't just assume by intuition that a particular approach will definitely work.

Each Component in the Architecture Has Its Role

Breaking the system apart, each component has its own responsibilities and evaluation focus:

• Prompt templates: Manage versions of instructions, examples, and context assembly; evaluate task success rate and wording sensitivity
• Document ingestion and retrieval: Maintain source versions, access filters, indexes, and excerpts; evaluate coverage, freshness, ranking, and citation support
• Tool execution: Fetch authoritative values, perform authorized actions; evaluate structure, units, timestamps, errors, and authorization
• Deterministic application logic: Enforce explicit business rules and output constraints; evaluate edge cases and reproducibility
• Training and model registry: Track datasets, base versions, fine-tuned artifacts, and evaluation evidence; evaluate target task improvement and other-task regression
• Orchestration and monitoring: Assemble context, route requests, log diagnostic information; evaluate end-to-end quality, cost, latency, and failure recovery

The source material gives a hypothetical policy assistant example: use retrieval to provide applicable document versions and citations, use an API to fetch live rates, use code for precise threshold calculations. The model can explain results, but rules that must be enforced belong in application logic.

Architecture Decision Record (ADR) is a lightweight engineering documentation practice for recording the context, options, final choice, and consequences of an important technical decision. Its value isn't in "recording what's correct," but in letting the team trace back later: why this was chosen at the time, what the constraints were, and what should be re-evaluated if conditions change. ADRs are typically short, numbered sequentially, and version-controlled alongside the codebase. The two ADRs given later in the article are examples of this format: one about retrieving changing policy documents, one about evaluating fine-tuning to stabilize classification behavior. Both explicitly state the decision's context, the choice, and the operational costs to bear—rather than merely claiming the approach works.

Diagnostic Helper Code (Educational Use)

The source material includes a Python educational snippet that maps already-diagnosed categories to candidate interventions. It doesn't perform diagnosis, authorize actions, or determine compliance:

from enum import Enum

class Finding(Enum):
    UNCLEAR_INSTRUCTIONS = "unclear_instructions"
    DOCUMENT_GAP = "document_gap"
    LIVE_VALUE = "live_value"
    EXACT_RULE = "exact_rule"
    STABLE_TASK_GAP = "stable_task_gap_after_baselines"

INTERVENTIONS = {
    Finding.UNCLEAR_INSTRUCTIONS: "evaluate_prompting",
    Finding.DOCUMENT_GAP: "evaluate_rag",
    Finding.LIVE_VALUE: "query_authoritative_tool",
    Finding.EXACT_RULE: "implement_tested_code",
    Finding.STABLE_TASK_GAP: "evaluate_fine_tuning",
}

def candidate_interventions(findings: list[Finding]) -> list[str]:
    """Return unique candidates in the order the findings were supplied."""
    if not isinstance(findings, list):
        raise TypeError("findings must be a list of Finding members")
    candidates = []
    for finding in findings:
        if not isinstance(finding, Finding):
            raise TypeError("each finding must be a Finding member")
        candidate = INTERVENTIONS[finding]
        if candidate not in candidates:
            candidates.append(candidate)
    return candidates

The candidate names deliberately describe "the next engineering action," rather than promising "this will definitely fix it." A real investigation may produce multiple categories at once, so the function accepts multiple findings.

Two Architecture Decision Records

The source material provides two hypothetical Architecture Decision Records (ADRs) demonstrating trade-offs in different scenarios.

ADR 001: Retrieving changing policy documents. The context is that the assistant needs to explain policy documents that change independently of model releases. The decision is to maintain a searchable source collection with versions and effective dates, refresh according to freshness requirements, apply access filters, retrieve relevant passages, and preserve citation metadata. The consequence is that document updates take effect without retraining the model, but ingestion and retrieval bring additional operational work and latency, and a new index alone can't prove answers are correct, complete, or compliant.

ADR 002: Evaluating fine-tuning to stabilize classification behavior. The context is that the classifier still makes recurring errors after checking task definition, labels, prompts, and context. The decision is to compare fine-tuning candidates against an improved baseline using representative, disjoint evaluation data, track the base model, training data, and fine-tuning configuration, include regression cases beyond the target categories, and maintain a rollback path. The consequence is that fine-tuning may improve stable task behavior, but the improvement must be proven; training data, evaluation, and model lifecycle management all add cost.

Evaluation and Maintenance Checklist

The source material concludes with an actionable checklist suitable for direct inclusion in team engineering standards:

• Version-control prompts, model configurations, data sources, and expected outputs to ensure reproducible comparisons
• Measure end-to-end task success rate with representative samples, looking at both averages and individual failure cases
• For RAG, evaluate source coverage, retrieval relevance, document versions, access control, and whether citations support each factual claim
• For tool calls, verify structure, units, timestamps, and error states; clarify when cached results are usable; don't silently use stale data when live values are required
• Test deterministic rules against boundary values and invalid inputs
• For fine-tuning, keep training and evaluation data separate, compare against the strongest practical baseline, and check for regression and drift before deployment
• Measure latency and cost per successful task based on actual application, with thresholds from your own requirements rather than invented generic benchmarks
• Diagnostic records should only retain information needed for the investigation, with proper retention policies and data handling

Start from evidence, choose candidate changes, measure results. Let retrieval, tool execution, rule enforcement, and learned behavior each be accountable to their own tests—that's the path from "uncertain" to "operable."

References

• Microsoft: Retrieval augmented generation and indexes
• Hu et al.: LoRA: Low-Rank Adaptation of Large Language Models (arXiv:2106.09685)