With Temperature Set to 0, Regex Beat the Prompt

While building a small offline tool that automatically converts WhatsApp messages into an order ledger, developer Khushighosh recorded three unexpected findings. These findings not only changed her development workflow but also provide a vivid case study in LLM engineering practice: when model outputs are sufficiently deterministic, the value of traditional software engineering methods re-emerges.

Deterministic Output Turns Prompt Engineering into A/B Testing

The first surprise: with the same model, same prompt, and temperature: 0, every re-run produced byte-for-byte identical results—including the failure cases.

This sounds like common sense, but in practice many developers are used to the randomness of LLM outputs and treat "retry a few times" as a debugging method. In this project, however, failures reproduced consistently. According to the author, this directly turned prompt tuning from "mystical parameter fiddling" into a quantifiable A/B test: change the prompt, run it once, and compare whether the failure patterns changed.

This observation has broad implications for AI application development. When a task demands high determinism (such as structured data extraction), a fixed temperature combined with fixed inputs lets you debug the LLM as a deterministic function. Prompt iteration now has a control group and an experimental group.

temperature is a parameter in the LLM sampling process that controls the randomness of output. In a softmax-based probability distribution, the temperature value scales the logits: the lower the temperature, the sharper the probability distribution, and the more the model tends to choose the highest-probability token; as temperature approaches 0, sampling degenerates into greedy decoding, i.e., choosing the most probable token at each step. Therefore, at temperature: 0, given the same model weights and the same input token sequence, the decoding path is fully deterministic, and the output is naturally byte-for-byte identical.

It is worth noting that this determinism has prerequisites: the inference backend (such as the Ollama runtime) must not introduce additional randomness, such as floating-point accumulation order changes caused by dynamic batching, non-deterministic GPU operators, or tokenizer version differences. In local single-request scenarios this can usually be satisfied, but it may not hold in cloud batch APIs. This is also why the author could reliably reproduce failure cases in an offline environment—determinism is the result guaranteed jointly by "same model + same input + same runtime," not by the temperature parameter alone.

Regex Is More Reliable Than Prompts

The second surprise is more intriguing: a single regular expression beat a carefully designed prompt.

Although the original text is truncated here, context suggests that when extracting order information from WhatsApp chat logs, the developer found that certain patterned content—such as amounts, dates, and product numbers—was more stable when matched with regex than when letting the model "understand and output" it. The model might occasionally miss fields or drift in format, whereas regex captures fixed patterns deterministically at zero cost.

This is not a denial of the LLM's value. On the contrary, it points to a more mature engineering approach: give deterministic tasks to deterministic tools, and give fuzzy semantics to the model. For example, intent classification of chat messages and normalization of unstructured descriptions are suitable for the model; while strongly patterned fields such as phone numbers, order numbers, and currency symbols are better handled by regex or parsers. Only by mixing the two can you balance robustness and development efficiency.

Structured output constraints (structured output / constrained decoding) are another engineering approach to format drift. The basic idea is to impose grammatical constraints on token selection during decoding: for example, through JSON Schema or regex grammars, the set of tokens allowed to be sampled at each step is restricted to those conforming to the target format, thereby guaranteeing valid output at the generation level. Mainstream implementations include llama.cpp based on GBNF grammars, Ollama's format parameter (which accepts JSON Schema), and OpenAI's Structured Outputs.

Compared with "writing a prompt that asks the model to output JSON," constrained decoding turns format correctness from a probabilistic problem into a certainty. But it only guarantees syntactic validity, not semantic correctness—field values can still be extracted incorrectly. Therefore, in tasks like order extraction, constrained decoding is suitable for securing the output structure, regex or parsers are suitable for securing strongly patterned fields, and prompts are responsible for intent understanding and ambiguity resolution—each playing its own role.

Practical Trade-offs of Offline Operation and Open Models

The project is called Order Ledger, positioned as a side-project tool that "runs on open models and can operate offline," with a tech stack involving Ollama. The author submitted it to the Hacktoberfest Open-Source AI Challenge on Dev.to, with tags including ollama and ai.

Choosing a local open model means giving up the convenience of cloud APIs in exchange for data never leaving the local machine, zero invocation cost, and offline usability. For data containing customer information, such as WhatsApp orders, privacy is a real selling point. The cost is the model's capability ceiling, inference speed, and deployment complexity. All three of the author's findings occurred under this constraint—precisely because the model's capabilities are limited and its output deterministic, engineering methods are all the more necessary to compensate for shortcomings.

Ollama is a local large-model runtime that wraps model download, quantized loading, and inference services, exposes an OpenAI-like HTTP interface, and supports custom Modelfiles to set the system prompt, temperature, context length, and other parameters. It typically relies on llama.cpp for inference underneath, so it can run 7B–14B quantized models on consumer CPUs or a single consumer GPU.

For a side-project tool, Ollama's value lies in simplifying "deploying a model" into "pulling and running a model," saving environment configuration and inference framework adaptation. But it also means developers must accept the precision loss from quantization, limited context windows, and single-machine throughput bottlenecks. Only by understanding this technical boundary can you reasonably judge which tasks are suitable for local models and which still need larger models or rule systems as a fallback.

Implications for AI Application Development

Stringing these three findings together yields a practical development principle: test determinism first, then talk about prompts. Before writing complex prompts, run the fixed input several times to see whether failures are stable. If they are, the problem lies in task design or tool selection, not in prompt wording. At this point, introducing regex, parsers, or structured output constraints often works faster than repeatedly revising the prompt.

Another implication concerns evaluation. When outputs are reproducible, you can build a small regression test set: run it after each change to see whether failure cases decrease and whether new failures are introduced. This test-driven prompt development is far more reliable than tuning by feel.

Finally, the offline route with open models suits data-sensitive and cost-sensitive scenarios, but it requires developers to break tasks down more finely. The model is not a universal glue; put it in the right place and leave the rest to traditional code—this may be the most valuable lesson to take away from this side project.

One-line Summary

At a fixed temperature, LLM outputs are deterministically reproducible; regex is more reliable than prompts; offline models need engineering to compensate for their shortcomings.

Paragraph Summary

While developing an offline tool that converts WhatsApp messages into an order ledger, the author recorded three unexpected findings: first, at temperature: 0, outputs from the same model and prompt are byte-for-byte identical, and failure cases reproduce consistently, turning prompt tuning into quantifiable A/B testing; second, regular expressions are more stable than carefully designed prompts for extracting strongly patterned fields such as amounts, dates, and order numbers, showing that deterministic tasks should be handed to deterministic tools; third, the project runs open models via Ollama and can be used offline, trading privacy and zero invocation cost for compromises in model capability and speed. From this the article distills the development principle of "test determinism first, then talk about prompts," and recommends building a regression test set to iterate prompts in a test-driven way.