Not Everything Is Worth a Token: When to Use AI and When to Use Deterministic Code

When AI Becomes an Expensive Hammer

In the wave of large language models (LLMs) sweeping through software development, a rather controversial trend is emerging: it seems that every feature, every data processing step, and every logical decision is being handed off to the model. A Hacker News discussion titled Not everything should cost a token raises a thought-provoking point—not every task requires calling an LLM; in many scenarios, deterministic traditional code is the better solution.

This trend has its historical context. The explosive popularity of ChatGPT in late 2022 gave millions of developers their first low-barrier access to powerful natural language processing capabilities. Compared to traditional NLP tools, LLMs require no labeled data and no training pipeline—just a single API call can accomplish what previously took months of engineering effort. This "low-friction" technical experience objectively gave rise to a cognitive bias: if a problem can be solved with an LLM, use an LLM first. This is especially evident among software startups—under pressure to quickly validate product hypotheses, "hand everything to the model" became the default path of least resistance, while technical debt and cost risks were deferred until the growth phase forced a reckoning.

The author's core argument can be summed up in a variation of an old saying: "When all you have is a hammer, everything looks like a nail." Today, many teams wield the LLM as an "all-purpose hammer," cramming problems that could be solved with a few lines of code, a regular expression, or a state machine into the model's context window—paying the price in Token cost, latency cost, and unpredictability cost.

The Hidden Cost of Token Economics

Every Call Costs Real Money

Handing a task to an LLM is far from free. Every API call is billed by Token—the longer the input and the more the output, the higher the cost. For high-frequency, large-volume tasks—such as format validation, data cleaning, and field extraction—routing everything through the model causes costs to grow linearly or even superlinearly with call volume.

The underlying logic of Token economics is worth elaborating here. A Token is the basic unit of measurement by which an LLM processes text, generated by tokenization algorithms such as BPE (Byte Pair Encoding)—it is not simply equivalent to a "character" or "word," but rather a subword unit that balances word frequency and word morphology. The BPE algorithm originated in the field of data compression; its core idea is to build a vocabulary by iteratively merging high-frequency character pairs: the algorithm first breaks all text down into individual characters, then repeatedly identifies the most frequently occurring adjacent character pairs and merges them into new subword units, until the vocabulary reaches a preset size. This process means that common words (like "the" and "is") are usually represented as a single Token, while rare words are split into multiple subword Tokens (e.g., "unbelievable" might be split into three Tokens: "un", "believ", and "able"). Typically, one Token corresponds to about 0.75 English words or roughly 1.5 Chinese characters, but punctuation, numbers, and code symbols are often split into multiple Tokens, causing the actual Token consumption of technical documents, structured data, and similar content to far exceed intuitive estimates.

It's especially worth noting that different Tokenizer strategies produce significantly different Token consumption—OpenAI's tiktoken, Google's SentencePiece, and the various tokenizers in the Hugging Face ecosystem can differ by 20%-40% in how they split the same text. This reflects fundamental divergences in tokenizer design philosophy: tiktoken tends to use Byte-Level Fallback for Unicode characters, ensuring the vocabulary's complete coverage of any input; SentencePiece provides a unified framework for unsupervised subword segmentation, supporting both BPE and Unigram algorithms, making it better suited for multilingual scenarios; while tokenizers optimized for code (such as the tokenization scheme used by CodeLlama) give more refined treatment to common symbols and keywords in programming languages. This difference is a systematically overlooked pitfall when designing cross-model cost comparisons or migrating between model providers. It's advisable to incorporate a "Tokenizer difference coefficient" into your cost estimation model at the design stage.

From a pricing structure standpoint, mainstream models like GPT-4o and Claude 3.5 Sonnet bill input and output Tokens separately, and output Tokens are typically 3-5 times more expensive than input Tokens—meaning the cost of scenarios where you "have the model output detailed results" is amplified further. Prices range from a few dollars to tens of dollars per million Tokens, and high-end reasoning models (such as o1 and Claude 3 Opus) can cost more than 10 times the rate of lightweight models. For enterprise-grade applications, a seemingly simple data processing pipeline processing a million records daily could easily push monthly Token costs past tens of thousands of dollars. This "pay-as-you-go" characteristic dictates that the higher the task frequency and the larger the data volume, the harder it becomes to ignore the marginal cost of using an LLM. It's worth noting that some cloud providers offer a Batch API mode that trades higher latency for lower prices, but this only mitigates rather than cures the cost problem of high-frequency calls.

More critically, such tasks often have clear rules and definite answers. Determining whether a string is a valid email, converting a date from one format to another, routing requests according to fixed rules—these problems were perfectly solved by computer science decades ago, and their execution cost is nearly zero.

Latency and Uncertainty: Two Major Engineering Pain Points

Beyond monetary cost, LLMs introduce two engineering challenges:

• Latency: A single model inference easily takes hundreds of milliseconds or even seconds, whereas deterministic code typically executes in microseconds.
• Uncertainty: Even with a low temperature setting, an LLM's output may still hallucinate, deviate in format, or produce occasional errors. For critical paths that require 100% reliability, this uncertainty is unacceptable.

The fundamental difference between deterministic programs and probabilistic models is rooted in different branches of computation theory. A Deterministic Program is a direct embodiment of the Turing machine model—the same input necessarily follows the same computational path and produces the same output; its behavior is fully predictable, testable, and debuggable. Regular expression engines, finite state machines (FSMs), and SQL query optimizers all fall into this category, and their correctness can be formally proven through mathematical induction.

LLMs are entirely different: they are essentially probability distribution estimators based on the Transformer architecture, and their output is in fact a probabilistic sampling of the "next Token." Understanding this requires delving a bit into the Transformer's inference mechanism: at each generation step, the model computes a weighted distribution over the entire context via the Multi-Head Self-Attention mechanism, transforms it through a feed-forward neural network, outputs a Logits vector over the vocabulary, and then normalizes it into a probability distribution via Softmax. The temperature parameter essentially scales this Logits vector—the lower the temperature, the sharper the probability distribution, and the more likely high-probability Tokens are selected; at temperature 0, it degenerates into Greedy Decoding, selecting the highest-probability Token at each step. Yet even so, this only guarantees determinism within a single model version—updates to model weights, version differences in the underlying CUDA kernels, and even precision differences in hardware floating-point operations (such as the accumulation of rounding errors between FP16 and BF16) can all introduce Output Drift.

What makes it more complex is that LLM providers usually do not give advance notice of model version changes, meaning a prompt that passes testing today may produce drastically different output three months later due to a silent model update. This opacity has, in engineering practice, given rise to a new class of practice—LLM Regression Testing: teams maintain a set of "gold standard" input-output pairs (a Golden Dataset), automatically running them after each model upgrade or prompt change to detect significant shifts in the output distribution. Some teams also introduce semantic similarity metrics (such as BERTScore and BLEURT) to replace exact string matching, in order to more robustly capture functional regressions rather than superficial changes in form. In scenarios with strong constraints on result consistency—such as financial risk control, medical diagnosis, and legal compliance—this unpredictability constitutes not only an engineering risk but may also trigger substantive regulatory compliance issues. Some industries' audit requirements demand that a system be able to fully reproduce the computational process of historical decisions, a requirement that probabilistic models are inherently ill-suited to meet.

Deterministic programs are precisely the opposite: given the same input, they always yield the same output—testable, verifiable, and debuggable.

When to Use AI and When to Use Code

Scenarios Suited for LLMs

The true advantage of large models lies in handling ambiguity, unstructured data, and open-ended problems:

• Natural language understanding and generation (summarization, translation, dialogue)
• Extracting semantic information from messy text
• Tasks requiring "common sense" or contextual reasoning
• Judgments where rules are difficult to enumerate and boundaries are blurry

In these domains, writing traditional rule-based code is either impossible or prohibitively expensive, and the generalization ability of LLMs brings a qualitative leap. Take sentiment analysis as an example: traditional methods require manually maintaining a sentiment lexicon containing tens of thousands of entries, and they are helpless in the face of neologisms, internet slang, and sarcastic expressions; whereas LLMs, drawing on language patterns learned from massive corpora, can understand subtle shifts in emotional nuance at near-human levels. This "Emergent Capability" is precisely the core value that distinguishes LLMs from all traditional NLP tools.

Emergent capability refers to complex abilities that a system spontaneously exhibits—abilities not explicitly required by the training data or optimization objective—once the model's parameter count crosses a certain critical scale, such as Few-Shot Learning, Chain-of-Thought Reasoning, and cross-lingual transfer. This phenomenon was systematically documented by research teams at DeepMind and Google in the 2022 paper Emergent Abilities of Large Language Models, which found that certain abilities exhibit a nonlinear "phase transition" characteristic: they are nearly zero below the threshold parameter count and rise sharply once the threshold is exceeded.

However, this picture is not without controversy. A 2023 follow-up study from Stanford University raised fundamental doubts about the discontinuity of "emergence"—researchers found that when using finer-grained evaluation metrics (rather than coarse-grained discrete accuracy), the improvement in capability is actually smooth and gradual, and the "phase transition" is to some extent an artifact of the choice of evaluation metric rather than a genuine sudden change in model capability. This academic debate reveals a deeper methodological issue: how we measure intelligence directly affects our judgment of "when" intelligence emerges. For engineers, this means that when citing "emergent capability" as a basis for technology selection, one must run benchmark tests for the specific task rather than directly applying the generalized conclusions of academic papers to production environments—a model that demonstrates emergent capability on the BIG-Bench evaluation may perform mediocrely in your specific business scenario. It is precisely this critical understanding of capability boundaries that transforms the judgment of "when to use an LLM" from intuition to engineered decision-making.

Scenarios Suited for Deterministic Code

For tasks with clear rules and predictable results, traditional solutions are almost always the better choice:

• Data format conversion and validation
• Mathematical computation and statistics
• Routing and branching based on fixed conditions
• Parsing and querying of structured data

A rational engineering practice is: use deterministic code to handle deterministic problems, and reserve the LLM's compute for the steps that truly require intelligence. Many mature AI application architectures (such as Agent workflows) also confirm this—let the model handle "decision-making" and delegate "execution" to reliable tool functions.

Hybrid Architecture: A More Mature Direction for AI Engineering

Behind this discussion points a more mature AI engineering paradigm: Hybrid Systems. In this architecture, the LLM is no longer the sole compute engine but rather the "brain" or "orchestrator" of the entire system—it decides when to call which deterministic tool, rather than performing all the low-level work itself.

This design retains the flexibility of large models in handling complex, ambiguous tasks while ensuring the system's reliability, predictability, and cost efficiency through deterministic components. The currently popular Tool Calling, Function Calling, and various Agent frameworks are all essentially engineering embodiments of this concept.

The technical implementation of hybrid architecture involves engineering design at multiple levels, and understanding its principles helps make better architectural decisions. Tool Calling is a core capability of today's mainstream LLM APIs: developers pre-register a set of function names, parameter signatures, and functional descriptions with the model. During inference, if the model determines that it needs external information or must perform a specific operation, it declares a structured "call intent" (usually in JSON format) in its output. The host program is responsible for actually executing it and injecting the result into the conversation context, after which the model continues reasoning based on the tool's return value. This mechanism decouples "language understanding" from "code execution" at the architectural level.

Agent frameworks such as LangChain, LlamaIndex, AutoGen, and CrewAI systematize this mechanism, building reusable layered component systems of "LLM decision-making + tool execution." At the academic level, this paradigm corresponds to the ReAct (Reasoning + Acting) framework proposed by Google in 2022—the model alternates between Chain-of-Thought Reasoning and tool-calling actions (Acting), forming an iterative perceive-reason-act loop. The core insight of ReAct is: keep language understanding and logical reasoning at the model layer, while delegating deterministic operations such as computation, database queries, format validation, and code execution to a reliable, efficient code layer.

From a systems engineering perspective, this decoupling also brings an important maintainability advantage: the deterministic tool layer can be unit-tested and integration-tested independently, and its behavior can be verified with all the quality assurance methods of traditional software engineering; while changes to the LLM layer (such as switching model providers or upgrading model versions) only require re-evaluating the decision routing logic, without worrying about the correctness of the underlying tools. This application of the "Separation of Concerns" principle in AI engineering is precisely the core advantage of hybrid architecture over end-to-end pure-model approaches in terms of engineering maintainability. Measured data shows that this hybrid architecture, compared to a "pure LLM" approach, not only reduces Token consumption by 40%-70% but also makes the entire system's behavior easier to audit end-to-end and debug step by step—because the input and output of every tool call is a recordable, replayable deterministic event.

It's worth mentioning that the design of hybrid architecture is not without its own engineering complexity. The proliferation of registered tools can give rise to the "Tool Selection Hallucination" problem—when the number of available tools exceeds a certain amount (empirically around 20-30), the model's accuracy in tool selection noticeably declines; increasing the depth of the tool-calling chain also accumulates the risk of error propagation, where the failure of a single tool can cascade to affect the entire reasoning chain. Therefore, mature hybrid architecture engineering practice must also introduce defensive designs such as timeout circuit breakers, result validation, and fallback strategies for tool calls, rather than treating "delegate to tools" as a once-and-for-all solution. In addition, the cumulative latency effect of each step in a multi-step tool-calling chain cannot be ignored: if an Agent task requires sequentially calling 5 tools, each LLM inference taking 1-2 seconds, the total end-to-end latency may exceed 10 seconds, which is often unacceptable in real-time interactive scenarios. Engineers need to make fine-grained trade-offs among the granularity of task decomposition, the possibility of parallel tool calls, and the latency budget of the user experience.

Three Pragmatic Suggestions for Developers

Although the original article did not generate much discussion on Hacker News, it touches on a widely overlooked problem in AI application development: over-reliance on LLMs. In an atmosphere where "everything must mention large models," calmly evaluating whether a task truly requires intelligence has ironically become a scarce engineering quality.

This scarcity has structural causes. On one hand, the extremely low barrier to using LLM APIs suppresses engineers' motivation to think deeply about "whether we should use it"; on the other hand, in product narratives that tout "AI-native" as a selling point, the technology choice itself carries a business-signaling function beyond the engineering domain—sometimes the value of using an LLM lies not in what problem it solves but in the market message it conveys: "we are an AI company." Recognizing and resisting this cognitive bias is one of the important capability dimensions that distinguish senior engineers from junior ones.

This phenomenon is known in the sociology of technology as the "Technology Signaling Effect"—technology selection is not only an engineering decision but also a narrative tool for conveying capability positioning to investors, customers, and the market. Similar phenomena have appeared many times in history: during the "big data" wave of the early 2010s, many data analysis tasks that could have been handled with Excel were forcibly migrated to Hadoop clusters; during the blockchain craze, numerous systems that did not need decentralized trust mechanisms were dressed up with "on-chain" architectures. Every wave of technological hype gives rise to this pattern of "technology over-use."

The deeper mechanism is that technology selection decisions are often not made by those who directly bear the engineering cost—product managers, investors, or marketing teams push the "AI-ification" narrative, while the engineering team has to bear the additional complexity and operational costs that result. This mismatch between decision-making authority and cost-bearing is the fundamental reason the technology signaling effect persists. From an organizational behavior perspective, breaking this mismatch requires two structural conditions: first, attributing LLM API costs directly to the business line that initiated the calls (rather than having the infrastructure team bear them collectively), so that decision-makers feel real cost feedback; second, introducing an "alternative-solution justification" step into the Tech Review process, requiring proposers to explain why traditional solutions are unsuitable. Only when these two mechanisms work together can systemic resistance to technology over-use be established at the organizational level. Recognizing which stage of which technology wave we are currently in is precisely one of the core judgment abilities of a technology leader.

The following three suggestions are worth bookmarking by every developer:

1. First ask yourself: does this problem have a definite answer? If so, prioritize traditional code.
2. Quantify the Token cost. In high-frequency scenarios, even if a single call is cheap, the cumulative cost can be substantial. It's advisable to build a cost model at the system design stage, estimating peak expenses by multiplying daily/monthly call volume by average Token consumption, and setting a cost-ceiling alert for each LLM call point.
3. Treat the LLM as a last resort, not a first choice. Let it focus on the things that only it can do well. Specifically, add a check to the Code Review process: for every newly added LLM call, require the developer to explain "why this task cannot be implemented with deterministic code"—this simple question can often filter out a large number of unnecessary model calls.

Ultimately, the essence of engineering is using the right tool to solve the right problem. The LLM is a powerful technology, but powerful does not mean omnipotent. Identifying which tasks are "not worth a Token" is precisely the first step toward building efficient, economical, and reliable AI systems.

Key Takeaways