Five Core Concepts to Learn Before Building LLM Applications

If you can build regular software, you can start building large language model (LLM) applications. But don't rush to open the API console just yet. This article outlines five core concepts to learn in order: machine learning, the nature of LLMs, the difference between generative and predictive AI, prompt engineering, and evaluation. Each concept comes with a hands-on exercise to help you build solid intuition before relying on vendor SDKs.

1. Machine Learning: Rules Learned from Data

In traditional software, programmers write the rules. Machine learning flips this: you show the model a large number of examples, and the training process adjusts the model until its predictions match the examples. The rules ultimately live in learned parameters, not in a readable document. Deep learning is the branch of machine learning that uses multi-layer neural networks and underpins almost all recent AI advances. Large language models are deep learning models trained on text.

One key practical consequence: a model's performance depends on the data it learned from, and it can confidently make mistakes in situations not covered by its training data.

Hands-on exercise: First write a rule-based fraud detection function (amount exceeds a threshold and comes from a new country), then replace the rules with a weighted sum using manually set weights. You're not training a model, but you get an intuitive feel that "decisions" are essentially numbers, and those numbers only make sense relative to examples.

Parameters are adjustable numerical values inside a model, typically weights and biases in a neural network. During training, optimization algorithms (such as gradient descent) repeatedly fine-tune these parameters based on prediction error, bringing the model's output closer to the targets in the training data. Parameter count is often used to measure model size—for example, "70B" means about 70 billion parameters. Parameters don't directly correspond to human-readable rules; instead, they encode statistical patterns in the data in a distributed way. This is why model behavior is hard to explain rule by rule, and one root cause of its potential for "confident errors."

Training/validation/test set splitting is a fundamental practice for evaluating a machine learning model's generalization ability. The training set is used to adjust parameters, the validation set to select hyperparameters and early stopping, and the test set only for final evaluation to estimate performance on unseen data. If the three are mixed, the model may overfit the training data, leading to degraded performance in real scenarios. This principle applies equally to LLM applications: examples used for prompt engineering debugging should not overlap with the final evaluation set.

2. What Exactly Is an LLM Doing?

A large language model treats text as tokens and predicts the next token based on everything before it. Repeat this process and you get a piece of text. "Large" refers to the scale of the neural network parameters and the amount of text learned during pre-training. Post-training teaches it to follow instructions and hold conversations.

Because the model is built to generate plausible text, it can be both fluent and wrong. The model itself doesn't know about events after its knowledge cutoff, nor does it know your private data. Most useful products wrap more around the model: retrieved documents, tools, memory, and checks.

Dan Klein, co-founder and CTO of Scaled Cognition, argues that people are using "plausibility engines": they assemble plausible output token by token. This is a great way to get prototypes fast, but it's hard to guarantee behavior.

Hands-on exercise: Give yourself a four-word vocabulary and a function that returns the highest-scoring next token. Start from a fixed prefix and loop ten times. Read the resulting string and note that no step in the loop checks the sentence against a source of truth.

Token is the basic unit of text that an LLM processes; it can be a word, subword, or character. A tokenizer splits raw text into a sequence of tokens, and the model then computes a probability distribution over the next token at each position. Different models use different tokenization schemes, which affects context window utilization and performance on certain tasks (such as arithmetic and handling rare words). Understanding tokens helps explain why models may perform poorly on character-level operations (like reversing a string), and why APIs charge by token.

Knowledge cutoff refers to the point in time when a model's pre-training data was collected. Events that happened afterward, newly emerged facts, or private data are not in the model's parameters. The model cannot actively perceive information after the cutoff unless external knowledge is injected at inference time through retrieval-augmented generation (RAG) or tool calls. When building applications, you must clearly identify which questions rely on the model's internal knowledge and which require external data sources, and design fallback strategies for time-sensitive content.

3. The Difference Between Generative AI and Predictive AI

Before 2022, most AI in business was predictive: the model looks at input and returns a label or number, such as fraud or not, demand forecast. Generative AI produces new content: paragraphs, images, songs, code blocks. Large language models generate text; image, audio, and video models generate media; multimodal models handle multiple types at once.

Generative AI and agentic AI are related but different. Generative models produce content when asked; agents use generative models to plan and take actions, calling tools and checking results along the way. It's advisable to get comfortable with generators before trying to build agents.

Hands-on exercise: Take a short customer support ticket and return two values from two separate functions. One is a label (such as billing or access), which is the old prediction task; the other is a drafted reply, which is the generation task. If you combine both in one return value, you'll start treating a category guess as the final answer.

Agent is a system built around a generative model that can not only generate text but also plan steps according to goals, call external tools (such as search, calculators, APIs), observe results, and iterate. Unlike single-shot generation, an agent typically includes components such as loop control, tool selection, memory management, and error handling. An agent's reliability depends heavily on the quality of the underlying generative model and the evaluation mechanism, so it's essential to first understand the basic behavior and limitations of generative models before building agents.

4. Prompt Engineering: Wording, Examples, and Format

Prompt engineering is the craft of the instruction itself: how to phrase the task, what examples to include, what format to require, how to structure it. Clearer prompts produce better outputs. It's the first lever everyone uses, and it still matters.

But don't treat clever wording as the winning move for your system. As models get better at understanding intent, wording stops being a lasting advantage and becomes a basic threshold. The work that used to hide in slogans now shifts to the documents, history, and tools you provide, and to what the system remembers across tasks. You'll learn these names later: context and memory.

Dan Klein describes prompting as a control surface, which fits its limitations: prompts have no precise semantics and no guarantees. If the model ignores you, rewording, capital letters, and extra exclamation marks are weak measures.

Hands-on exercise: One task, three prompts. A bare instruction; the same instruction plus one example; the same instruction plus a required format (for example, a JSON object with two fields). Save all three outputs. You're practicing wording, examples, format, and structure—that's the whole craft above.

5. Evaluation: Consistency Matters More Than Average Accuracy

Dan Klein argues that "how many scenarios did I get right" is the wrong question for launch. What he wants to know is: for how many scenarios is the system right every time, including a hundred times in a row. In his view, a system that is about 80% or 90% correct is still unshippable if one in ten customers is told a flight is booked when it isn't. A system that is 70% correct every time is more valuable because that consistency can ship in a product.

He also argues that adding a second model on top to check the first is a weak fix. Both can be wrong, and often fail on the same hard cases, so he estimates 80% checking 80% ends up around 82%. Adding more checkers makes the stack expensive and slow.

Your first check can be smaller than a benchmark suite: freeze the input, run the call multiple times, record whether the output is consistent and whether it matches facts you know. Fluency is not this check.

On the reading list, the AI, decoded page is divided into five levels. If you're building, skim Level 1 (how LLMs work, why they cost what they do, why they make things up), then skim Levels 4 and 5, where evaluation comes up as a builder's concern. After that, use Start here: five episodes on agents, evaluation, agent memory, AI infrastructure, and shipping AI in real products.

Benchmark is a standardized evaluation dataset and metric used to compare different models or systems on specific tasks. Common benchmarks include MMLU (multidisciplinary knowledge), GSM8K (mathematical reasoning), HumanEval (code generation), and others. However, a high benchmark score doesn't equal product reliability: benchmarks usually cover limited scenarios and may be contaminated by training data. For a specific application, building a custom evaluation set that reflects real usage scenarios and focusing on consistency (stable results across multiple runs of the same input) is more practically meaningful than chasing public benchmarks.

Practical Checklist

• Treat machine learning as examples and parameters, and expect confident errors outside the data.
• Understand a paragraph as repeated next-token prediction, accompanied by hallucinations, knowledge cutoff, and no private data unless you provide it.
• Keep predicted labels and generated drafts in separate functions.
• Practice wording, one example, and fixed format, then move to context and memory.
• Before trusting a single fluent answer, repeat a frozen input and compare with known facts.

This content is based on discussions from the Chain of Thought podcast and related glossaries; more resources can be found on its website.

One-line Summary

Before building LLM applications, master five concepts: machine learning, token prediction, the difference between generation and prediction, prompt engineering, and consistency evaluation.

Paragraph Summary

The article proposes that before calling vendor APIs, developers should understand five core concepts in order: machine learning learns rules through examples and parameters, and models can confidently err outside training data; an LLM is essentially repeated next-token prediction, fluent but possibly wrong and unaware of knowledge cutoffs and private data; generative AI produces new content while predictive AI returns labels or numbers, and agents plan actions on top of that; prompt engineering is a basic threshold rather than a lasting advantage, with context and memory as deeper levers; evaluation should focus on the proportion of scenarios the system gets right every time, not average accuracy—an 80% correct system may be unshippable due to inconsistency. Each concept comes with a hands-on exercise to build intuition.