Fixing the Three Big Weaknesses of LLMs: Knowledge, Computation, and Tool Use

Article

When we ask a large language model a common-sense question — "Is a higher score better in golf?" — it might confidently answer "Yes." But the reality is the exact opposite: in golf, fewer strokes is better. This seemingly simple mistake reveals a fundamental limitation of LLMs: what's stored in their "heads" isn't always reliable — no matter how carefully they "reason," the answer can still be wrong.

This limitation has a deep-rooted cause. LLMs store parametric knowledge — world knowledge compressed and encoded into billions of neural network weights, rather than stored in a precisely retrievable structured knowledge base. This compression is inherently lossy: low-frequency knowledge is easily forgotten, information after the training cutoff is completely missing, and certain common-sense facts (like golf rules) may be systematically skewed due to imbalanced positive and negative representations in the training corpus. In other words, a model's errors are often not random — they follow predictable patterns.

This is the fifth installment in our prompt engineering series. In the last issue, we covered self-consistency and chain-of-thought prompting — helping AI "reason more reliably" and break down complex tasks. But no matter how strong the reasoning ability, if the knowledge is flawed, the arithmetic is off, or the information is outdated, none of it matters. In this issue, we focus on three solutions: generated knowledge prompting, program-aided reasoning, and tool use — each targeting one of the three major weaknesses in LLMs: knowledge, computation, and external capability.

The Three Approaches: From Lightweight Prompts to Full Agent Architecture

Let's first get the relationship between these three approaches straight. They all address the same core problem: LLMs have gaps in their capabilities — how do we fill them? But the approaches are fundamentally different, ranging from lightweight to heavyweight along a clear spectrum.

Generated knowledge prompting is the lightest approach — pure prompt engineering with no external systems required. Two prompts are all it takes, making the barrier to entry extremely low. Program-aided reasoning requires connecting a Python interpreter — but note that it's an "executor," not a "knowledge source" — used to solve the problem of inaccurate computation. Tool use carries the highest engineering cost, requiring integration with search engines, calculators, translation APIs, and other external services, with the model deciding when to invoke them — but it also fills the most gaps.

These three paths target different levels of weakness: when computation is inaccurate, let code do the math; when knowledge is outdated, let a search engine look it up; when the model's internal knowledge is misfiring, have it write out the knowledge first. Which approach to use depends on where the weakness lies.

Approach 1: Generated Knowledge Prompting

When a model makes mistakes on common-sense questions, it's often not because it's "dumb" — it's because implicit knowledge was incorrectly or partially retrieved. The fix is elegant: instead of hoping the model automatically retrieves the right knowledge, have it first articulate what it knows, then answer based on that knowledge.

The process has two steps. In the first step, use few-shot examples to prompt the model to generate knowledge — provide a few "input → knowledge" demonstrations to guide it in producing relevant knowledge statements for a new question. Few-shot (few-shot learning) is one of the core capabilities of LLMs, systematically demonstrated by GPT-3 in its 2020 paper: by providing a small number of input-output examples in the prompt, the model uses in-context learning to understand the task pattern and generalize to new inputs without updating its weights — meaning the entire knowledge generation process happens at inference time, with zero training cost. In the second step, combine the generated knowledge with the original question in a prompt, and have the model answer based on that knowledge. Returning to the golf example: first generate "The goal of golf is to complete the course in as few strokes as possible," then answer — and the response flips from "Yes" to "No."

Lilian Weng calls this approach "internal retrieval" — as opposed to "external retrieval" which queries an external vector store, internal retrieval has the model query its own knowledge base, eliminating the engineering complexity of a vector database.

But there's a risk to watch out for: the generated knowledge itself might be wrong. If the model generates incorrect knowledge and then answers based on it, you get compounded errors. In production environments, you shouldn't use just one generated piece of knowledge directly — instead, sample multiple outputs, filter for quality, and discard obviously incorrect knowledge.

Approach 2: Program-Aided Reasoning (PAL / PoT)

LLMs have long been unreliable at arithmetic. Once decimals, division, and multi-step mixed operations are involved, error rates climb steadily. The root cause is simple: the model is not a calculator — every digit is generated token by token, not computed digit by digit.

In 2022, two teams independently arrived at the same solution — don't let the model calculate itself; have it write code and let the Python interpreter do the math. This became PAL (Program-Aided Language Models) and PoT (Program of Thoughts). PAL was proposed by Carnegie Mellon University, while PoT was independently published by the University of Maryland. Their core ideas are identical; the difference is that PoT places greater emphasis on mapping complex multi-step reasoning fully into program structures with loops and conditionals, while PAL focuses more on simple arithmetic scenarios. Experiments show that on math benchmarks like GSM8K, both approaches can improve accuracy by 10–20 percentage points compared to pure chain-of-thought.

The approach is straightforward: replace the natural language reasoning in a chain of thought with Python code. Instead of writing "4 multiplied by 30 equals 120," the model writes time = 30 * 4, feeds the code into an interpreter, and gets a precise result — not "probably correct with high probability."

The key insight here is decoupling "computation" from "reasoning": LLMs excel at semantic understanding (knowing that "4 times the duration" maps to multiplying by 4), while interpreters excel at precise computation (the output is always exact). Each does what it does best — math problems can be made essentially error-free.

The tradeoff is that you need a secure code sandbox. Running model-generated code directly on a production server carries serious security risks — the model might generate code that accesses the filesystem, makes network requests, or even executes system commands. In practice, you must use isolated environments: Docker containers with restricted system call permissions, dedicated code execution cloud services like E2B, or Python's RestrictedPython library, all paired with execution timeouts and memory limits. The model also needs a baseline ability to generate valid code.

Approach 3: Tool Use and ReAct

The first two approaches each have a ceiling: generated knowledge prompting can't handle breaking news, and program-aided reasoning can't handle translation, search, or calendar lookups. The core idea behind tool use is to acknowledge that LLMs aren't omnipotent, equip them with external tools, and let them decide when to call on external sources.

At the training level, there are two representative approaches. TALM has the model repeatedly interact with tool APIs — if it does well, those interactions are kept and used to expand the training set, loosely simulating reinforcement learning. Toolformer is more elegant — proposed by Meta AI in 2023, it uses a self-supervised approach: first use few-shot prompting to have the model annotate potential API call locations in text, then execute those calls and observe "whether adding the call result reduces the model's loss in predicting subsequent content" — if it does, the example is kept for fine-tuning. This mechanism of using "is it actually useful" as a selection criterion teaches the model to invoke tools only when truly needed, avoiding over-reliance on external APIs. Toolformer covers five categories of tools: calculators, question-answering systems, search engines, translation systems, and calendars — each filling a different gap.

But an important distinction needs to be made: TALM and Toolformer are "training-side" tool use approaches that modify model weights; whereas the more mainstream industrial approach is a "prompt-side" solution — no model modification, using prompts to guide the model in deciding when to use tools. Claude, for example, uses XML tags in the System Prompt to control "when to act proactively and when to wait for instructions."

The ultimate form of this logic is the "think-act" loop of ReAct (Reasoning + Acting, proposed by Google DeepMind in 2022). ReAct interweaves reasoning and action into a three-step loop: the model first thinks (Thought) to analyze the current state, then acts (Act) by invoking a tool, receives an observation (Observation), and continues thinking — until the task is complete. Building on this, industrial implementations like LangChain, AutoGPT, and Claude's Tool Use layer memory systems (short-term context window + long-term vector database), multi-tool routing, and error recovery mechanisms on top of the ReAct loop — ReAct is the core skeleton of today's AI Agents.

How to Choose: From Point Solutions to Agent Architecture

How do you decide which approach to use? A simple decision logic:

• Model keeps getting a specific answer wrong → Use generated knowledge prompting — lowest cost;
• Model keeps getting arithmetic wrong → Use program-aided reasoning — connect a Python interpreter;
• Model doesn't know the latest information → Use tool use — connect a search engine.

If multiple weaknesses exist simultaneously, go straight to tool use and configure multiple APIs. And when the model also needs to "autonomously decide when to use which tool," it evolves into a full Agent architecture: ReAct + tool chain + memory system.

More importantly, these approaches don't just "work together" — they naturally belong together: chain-of-thought handles reasoning, chained prompts decompose tasks, self-consistency votes to correct errors at critical steps, generated knowledge prompting compensates for knowledge gaps, program-aided reasoning takes over computation, and tool use handles all external capabilities. Together, they mark a shift — prompt engineering is no longer just about "how to talk to a model," but about "how to build a team around the model."

Summary: Three Approaches, Each with Its Role

• Generated knowledge prompting: When uncertain, study first — no external tools required, lowest overhead, but may hallucinate, so filtering is needed;
• Program-aided reasoning: Figure out the formula, but don't compute it yourself — LLM sets up the logic, Python produces the result, maximum precision, but requires a sandbox;
• Tool use: When you can't handle it yourself, call external sources — from search engines to calendar APIs, the model decides when to invoke them.

From "teaching AI to think" to "enabling AI to look things up, compute, and take action," we've made a critical leap — from vibe coding to harness engineering. In the next installment, we'll bring all of this into production and discuss the engineering deployment of prompt engineering.

Key Takeaways