Flagship Model vs. Local 27B: Which Follows Code Instructions More Faithfully?

Having LLMs Each Build a Game: A Fascinating Comparison Experiment

Recently, an intriguing set of experiments appeared on Reddit. The experimenter gave the same instruction—"Build a Call of Duty (CoD)-style game, and don't make any mistakes"—to two types of models: on one hand, Anthropic's flagship frontier model Claude Opus, and on the other, a 27-billion-parameter (27B) local open-source model that can run on a personal machine.

Claude Opus is the top-tier product in Anthropic's flagship large language model lineup. Anthropic was co-founded by Dario Amodei, former VP of Research at OpenAI, with a focus on AI safety research. Opus leads the industry in code generation, complex reasoning, and long-context understanding, but its API costs are relatively high, typically targeting enterprise users and professional developers. In contrast, the 27-billion-parameter local open-source model represents the mainstream scale of large language models that can currently run on consumer-grade hardware. Thanks to techniques like 4-bit quantization in the GGUF format, the VRAM requirement for running a 27B model can be compressed to around 16GB, truly making it possible to "run on your own machine."

A Bit of Background on Quantization: The GGUF format and 4-bit quantization are the technical cornerstones that made local large models widely accessible. Quantization is the process of compressing model weights from 32-bit or 16-bit floating-point numbers to lower precision (such as 4-bit integers), trading a small amount of precision loss for a dramatic reduction in VRAM usage. The llama.cpp project engineered this technique, reducing the VRAM requirement for a 27B-parameter model from roughly 54GB (FP16) to about 16GB (Q4 quantization)—right within the range available on mainstream consumer GPUs—truly opening the door to "running large models on personal machines." GGUF (GPT-Generated Unified Format), as the standard format of the llama.cpp ecosystem, packages model weights, quantization parameters, and metadata into a single file, greatly simplifying the local deployment process, and has now become the de facto standard for distributing quantized models in the open-source community. It's worth noting that beyond the GGUF/llama.cpp system, quantization schemes like ExLlamaV2 and AWQ (Activation-aware Weight Quantization) also offer competitive options for different hardware scenarios—AWQ identifies and protects the weights most critical to model performance by analyzing activation distributions, often achieving lower precision loss at the same bit width; ExLlamaV2, meanwhile, is deeply optimized for NVIDIA GPU Tensor Cores, giving it an edge in inference speed. These diverse quantization schemes together enrich the local deployment tech stack, allowing developers to flexibly balance VRAM capacity against inference speed requirements.

This was the experimenter's "second round." The previous day, he had the models replicate Grand Theft Auto (GTA), and this time switched to CoD, keeping the core instruction unchanged: "make no mistakes." A seemingly simple comparison, yet it unexpectedly exposed some subtle and thought-provoking phenomena in large model behavior.

The Expensive Flagship Model Actually "Cheated"

The most dramatic moment of the experiment came from the expensive frontier model Opus. When generating the CoD game, it took the initiative to add wallhacks—enemy health bars would render directly through the wall on the left side.

Wallhacking is one of the most common forms of cheating in shooter games, allowing players to see hidden enemy positions through walls. In competitive shooters like Call of Duty, this is considered a serious disruption to game balance and is a primary target of Activision's RICOCHET anti-cheat system. Notably, precisely because wallhacks are notorious within the CoD player community—countless forum posts, video comments, and Reddit threads discuss how to identify, report, and counter wallhacks—such vocabulary and behavioral descriptions appear with extremely high frequency in the model's training corpus. Opus incorporating it into the game prototype may reflect exactly a bias of "corpus frequency equals feature salience": when "recreating the CoD experience," the model mistakenly incorporated a high-frequency concept from its training data into its "typical feature set," misidentifying a widely discussed negative phenomenon as a component of the game.

The Deeper Mechanism of Training Corpus Bias: From an information-theoretic perspective, during pretraining a language model is essentially learning "which vocabulary tends to co-occur in which contexts." When "wallhack" and "Call of Duty" form a strong association across hundreds of millions of web texts, player forums, and YouTube comments, the model encodes this statistical association as implicit knowledge in its parameters. However, this co-occurrence relationship in reality is a "critical association" (players criticizing, reporting, and discussing how to counter cheating) rather than a "constitutive association" (the game itself being designed to include this feature). The existing pretraining paradigm lacks explicit modeling of association polarity—the model knows that "wallhack often appears together with CoD" but cannot automatically infer "this co-occurrence exists because players hate it." This "polarity blindness" can, in specific scenarios, quietly transform negatively-discussed subjects in the training corpus into positive constitutive elements in the model's generated content. For designers of RAG (Retrieval-Augmented Generation) systems, this phenomenon carries a similar warning: when the retrieved reference documents primarily come from negative case analyses, the generation module may mistake "counterexamples" for "examples."

From a cognitive science perspective, this behavior is essentially a manifestation of the large language model's "Prototype Effect." The so-called prototype effect refers to the model constructing conceptual prototypes through statistical co-occurrence relationships rather than genuine semantic understanding—the high-frequency co-occurrence of "wallhack" and "CoD" in the training corpus causes the model to also activate the strongly associated word "wallhack" when activating the concept of "CoD game." However, the model fails to correctly understand the nature of this co-occurrence relationship: in real-world context, the two are in an "oppositional association" (players hate and report cheating) rather than a "constitutive association" (the game itself contains this feature). This misjudgment of association direction is precisely an inherent limitation in causal reasoning of current language models trained on the Next-Token Prediction paradigm—the model excels at recognizing "what appears alongside what" but struggles to understand the semantic relationship behind "why they appear together."

The Extended Impact of the Prototype Effect: This phenomenon is not isolated in the field of code generation. When a developer asks a model to "generate a login form," some models may automatically add a "remember password" checkbox or even weak password hints—because these elements co-occur frequently in the login form examples of the training corpus, even if the user didn't request them. A deeper challenge is that the model cannot distinguish between "descriptive co-occurrence" (an article discussing a feature) and "normative co-occurrence" (a feature that should exist), whereas humans naturally invoke commonsense causal inference when processing such information. This is precisely the core deficiency of large language models at the "world model" level: they are extremely powerful pattern-matching engines, but lack intrinsic understanding of the real world's causal structure, and can only approximate such understanding through statistical regularities in massive training data. Part of the value of techniques like Chain-of-Thought prompting lies precisely in forcing the model to explicitly reason through intermediate steps, which to some extent alleviates this "polarity blindness"—giving the model the chance to explicitly judge the nature of an association before generating the final answer.

In other words, Opus believed that an "authentic CoD experience" should include cheat functionality. Yet the experimenter never raised this requirement at any point.

This detail is intriguing because it touches on the problem of large models "over-inferring intent." Frontier models are often trained to be "smarter" and more proactive, attempting to fill in needs the user didn't explicitly state. The root of this behavior lies in the current mainstream alignment method, RLHF (Reinforcement Learning from Human Feedback).

The Inherent Bias of RLHF: The RLHF (Reinforcement Learning from Human Feedback) training process typically involves three steps: first, fine-tuning the base model with supervised learning; then training a "reward model" to simulate human scoring preferences; and finally optimizing the language model with reinforcement learning algorithms like PPO to maximize the reward score. The key problem with this mechanism is that human annotators tend to give higher ratings to answers that "exceed expectations and provide additional detail," which invisibly and systematically encourages the model to make inferential completions—even when the user hasn't explicitly requested them. This incentive signal is especially pronounced in code generation scenarios, because "more complete code" often receives higher ratings in annotation than "code that just meets the requirements." A deeper problem is that the reward model itself can be contaminated by "annotator-pleasing" samples, forming the so-called "Reward Hacking" phenomenon—the model learns to maximize the reward signal rather than genuinely understand user intent. This is precisely the institutional root of behaviors like Opus adding wallhacks. Alignment methods that have emerged in recent years, such as DPO (Direct Preference Optimization) which requires no explicit reward model, encode human preference data directly into model parameter updates, bypassing the reward model intermediary and attempting to partially mitigate the "reward hacking" problem; while methods like Constitutional AI (CAI) try to reduce reliance on manual annotation by having the model self-critique and revise. But regardless of the method used, the tension between "over-completion" and "instruction fidelity" remains one of the core challenges in current alignment research.

This proactivity is a double-edged sword:

• On the positive side, it can provide more complete solutions than expected;
• On the negative side, it introduces things the user never wanted, and may even violate explicit constraints.

Ironically, in a task that explicitly required "no mistakes," the model itself introduced cheating behavior that can be considered a "mistake"—which is in itself a deviation from the instruction.

The 27B Local Model: The Honest "Little Guy"

In stark contrast was that 27B local small model. It dutifully produced a normal wall, without any superfluous wallhack functionality.

So we see a counterintuitive result:

The expensive flagship model became the "cheater," while the 27B model with far fewer parameters, running on the user's own machine, turned out to be the honest one, faithful to the instructions.

This doesn't mean the 27B model has stronger overall capabilities—it's very likely simply because its capability boundaries are more conservative, so it won't make excessive autonomous inferences. But from the dimension of instruction following, its performance is indeed closer to the user's original intent.

The "Conservative Inference" Capability Boundary Explanation: The reason small models perform more consistently in instruction following may partly be that their "ceiling of inference capability" is inherently lower—they lack sufficient parameter capacity to construct a complex "user intent model," so they can only execute instructions more literally. This phenomenon of "doing it right because it can't do more" has a similar observation in software engineering: simple tools are sometimes more predictable than complex frameworks. However, this doesn't mean that instruction-following ability is naturally opposed to model scale—through dedicated SFT (Supervised Fine-Tuning) data construction, large models can equally be trained into highly faithful instruction executors. The Instruct versions of series like Mistral and Qwen, precisely by carefully curating and constructing constraint-following training data, simultaneously maintain high instruction fidelity at large parameter scales, proving that "powerful capability" and "predictable behavior" are not mutually exclusive.

Instruction-Following Ability ≠ Overall Model Capability

This experiment points to an easily overlooked evaluation dimension: the strength of a model's capabilities and its degree of faithfulness to instructions are two different things.

Instruction Following is an independent evaluation dimension in the NLP field. Benchmarks specifically targeting this dimension include IFEval and FollowBench.

Benchmark Explanation in Detail: IFEval (Instruction Following Evaluation) is a specialized benchmark proposed by Google in 2023, containing over 500 programmatically verifiable instruction constraints, such as "the answer must be under 200 words," "no commas may be used," "must contain a specific keyword," etc., and uses automated scripts rather than human judgment to determine whether the model precisely adheres to the constraints, thereby avoiding the subjectivity of traditional human evaluation. FollowBench further introduces difficulty grading with multi-layered constraints, specifically testing model stability and consistency when constraints are stacked—experiments found that when constraints increase from 1 to 5, nearly all models experience a cliff-like drop in compliance rate, more pronounced in larger models. The findings of both benchmarks point to a common pattern: larger models perform better on complex reasoning, but sometimes underperform smaller models fine-tuned for the purpose when it comes to strictly following constraint-based instructions—a phenomenon researchers call the "alignment tax," meaning there is a certain inherent tension between capability improvement and instruction fidelity. It's worth noting that this nonlinear relationship is no accident—multiple studies show that instruction-following ability does not increase monotonically with parameter count, and small models that have undergone targeted SFT (Supervised Fine-Tuning) often outperform larger models that have only undergone general alignment in constraint-based instruction scenarios. This finding has prompted the industry to begin treating instruction-following ability as an independent optimization target, specifically constructing high-quality constraint-following training data rather than treating it as a natural byproduct of general capability improvement.

The Practical Significance of the "Alignment Tax": The concept of the alignment tax reveals a fundamental trade-off in model development: to make a model more "helpful" and more proactive in meeting users' implicit needs, researchers often need to inject more inferential completion tendencies through RLHF; and once this tendency becomes too strong, it erodes the model's precision in following explicit constraints. For enterprise users, this tension is especially real—system prompts in production environments often contain numerous strict constraints (output format, prohibited vocabulary, response length, etc.), and the model's "clever proactivity" may instead become the enemy of reliability. This is precisely why some leading enterprises, when selecting models, list "the stability and consistency of instruction following" as a primary evaluation dimension on par with "task completion quality."

When the industry evaluates large models, the focus is often on the capability ceiling—whether they can write more complex code and solve harder problems. But in actual production environments, "whether it can do exactly what I ask, no more and no less" is equally critical. A powerful model that "thinks it's being clever" can sometimes actually increase the cost of debugging and error correction.

Local Small Models Are Rapidly Approaching Frontier Levels

Beyond the "cheating" topic, the experimenter expressed another observation:

It always amazes me that a 27B model running on my own machine can come this close to frontier model levels on tasks like pure HTML and JavaScript.

This confirms the overall trend of the open-source model ecosystem in recent years. The rapid development of the open-source LLM ecosystem in recent years has largely relied on the synergy of several key technological breakthroughs: Meta's release of the LLaMA series laid the ecological foundation for open-source base models, and its open-weight strategy attracted researchers worldwide to perform instruction fine-tuning, domain adaptation, and quantization compression on top of it, forming a self-reinforcing open-source flywheel; efficient quantized inference achieved by projects like llama.cpp compressed models that originally required data centers to sizes runnable on consumer-grade hardware; and a new generation of efficient small models represented by Mistral 7B/8x7B, Qwen 2.5, and Gemma, by improving Transformer architectures (such as GQA grouped-query attention and sliding window attention) and substantially raising the quality of training data, proved that parameter count is not the sole determinant of performance.

Among these, Knowledge Distillation technology is key to understanding this catch-up effect. The idea behind knowledge distillation is to have a small model (the student model) systematically learn the output distribution of a large model (the teacher model)—learning not only the final answer given by the large model (hard labels), but also its probability distribution over all possible outputs (soft labels). These "soft labels" contain the teacher model's fine-grained judgments of semantic similarity: for example, in code completion tasks, the teacher model assigns similar probabilities to multiple semantically equivalent implementation approaches, and this distributional information is itself a form of high-density knowledge encoding, conveying the teacher model's implicit knowledge far more efficiently than a single "correct answer." Additionally, architectural innovations like GQA (Grouped-Query Attention) have substantially improved the inference efficiency of small models—GQA, by having multiple query heads share the same set of key-value heads, compresses the VRAM usage of the KV Cache to a fraction of its original size with almost no loss in model expressiveness, making it possible to deploy models with larger parameter counts under limited VRAM.

On structured tasks like web front-end code generation, where training corpora are extremely abundant on the internet, this catch-up effect is particularly significant: the syntax rules of HTML/CSS/JavaScript are highly structured, and the training corpus is large in quantity and uniform in quality, allowing even smaller models to achieve near-saturation task performance given sufficient training data.

The Insights of the "Capability Saturation" Phenomenon: On specific structured tasks, model capability exhibits a nonlinear "saturation curve" as parameter count grows—when training data quality is high enough and the task's inherent complexity is limited, even a difference of an order of magnitude in parameter count may result in a negligible gap in final task performance. This phenomenon has important implications for developers' selection decisions: for highly structured, corpus-rich tasks such as web front-end development, SQL query generation, and regular expression writing, prioritizing a local 27B model is often the better engineering decision; while for tasks requiring cross-domain knowledge integration, complex multi-step reasoning, or open-ended creation, the scale advantage of frontier large models truly comes into play. Understanding the "capability saturation point of a task" is the core cognitive framework for making a reasonable trade-off between local models and cloud APIs.

On structured, corpus-abundant tasks such as front-end code generation and common web development, medium-scale open-source models can already provide quite usable results. For developers, the advantages of local models are obvious:

• Cost: eliminates API call fees;
• Privacy: code and data never leave the local machine;
• Controllability: free to fine-tune and deploy, unaffected by changes in service provider policies.

The Engineering Maturity of Local Deployment: It's worth adding that the "controllability" advantage of local models has, at the engineering level, far surpassed the early proof-of-concept stage. Tools like Ollama, LM Studio, and Jan encapsulate the startup, switching, and API calls of local models into a Docker-like streamlined experience, allowing developers to call local models through the standard OpenAI-compatible API interface, achieving seamless switching between cloud and local with almost no modification to existing code. This "API compatibility" ecosystem building has substantially lowered the integration cost of local models in production environments, making the hybrid deployment strategy of "prioritizing local models for daily development tasks and only calling cloud APIs when necessary" truly engineering-feasible, rather than remaining at the level of a technical ideal. It's worth mentioning that high-performance inference frameworks like vLLM have further boosted the throughput of local models to near commercial-API levels—by efficiently managing KV Cache VRAM fragmentation through techniques like PagedAttention, vLLM's throughput in multi-concurrency scenarios can reach dozens of times that of naive implementations, making local deployment feasible in team collaboration scenarios as well, not just an exclusive tool for individual developers.

When a 27B local model is already "good enough" for daily web development tasks and behaves more predictably, its appeal to individual developers and small teams becomes very real.

Open-Source Project: codehamr

The experimenter open-sourced the relevant code to GitHub in a project named codehamr (repository: https://github.com/codehamr/codehamr), so anyone can use it and verify the experimental results.

This "open-source reproducibility" approach is commendable. The biggest problem with large model comparison experiments is their high subjectivity and difficulty in reproduction. Publishing the prompt, code, and results together allows the community to verify firsthand: Does Opus really add wallhacks? Just how close is the 27B model's output to the frontier? Open-source reproduction is not only a reflection of scientific integrity, but also an important way to drive the community toward consensus understanding of model behavior—compared to individual observation, reproducible public experiments more easily trigger broader verification and discussion, thereby distinguishing "incidental phenomena" from "systematic patterns." It's worth noting that such community-driven informal experiments have formed a unique complementary value in the field of AI capability evaluation: compared to standardized academic benchmarks, they can often more keenly capture a model's "personality traits" in real usage scenarios—those behavioral patterns that won't be exposed on carefully designed test sets but recur repeatedly in daily interaction.

The Methodological Value of Community-Driven Evaluation: Academic benchmarks (such as HumanEval, MBPP, GSM8K) pursue objectivity and reproducibility in their design, but this rigor also brings the concern of "benchmark contamination"—the model may have seen the test questions in its training data, leading to inflated benchmark scores. Community-driven novel tasks (such as "having the model replicate CoD") happen to avoid this problem: identical tasks rarely exist in the training corpus, so the model must genuinely reason and generate rather than retrieve from memory. Although such experiments lack statistical significance, they provide an observational window into the "behavioral ecology" that benchmark testing struggles to capture, complementing formal evaluation and together forming a three-dimensional understanding of model capabilities and limitations. Furthermore, "prompt reproducibility" itself is a dimension worth attention—since large model outputs are inherently random (controlled by the temperature parameter), even using the exact same prompt, different runs may produce different results. Ideal community experiments should disclose sampling temperature, Top-P, and other inference parameters, and perform multiple runs for sampling, in order to distinguish "the model's systematic behavioral tendencies" from "the incidental output of a single sampling."

Several Reflections From This Experiment

This seemingly lighthearted little experiment of "having LLMs make games" actually mirrors several industry signals worth noting:

1. A model's "proactivity" needs boundaries. The stronger the model, the more prone it is to over-inferring intent. While the RLHF training mechanism encourages models to provide "beyond-expectation value," it also sows the hidden risk of instruction deviation. How to strike a balance between "intelligent completion" and "faithful execution" is an important topic in large model product design. System Prompt Engineering and more refined expression of constraints are mitigation measures that developers can actively control at present.
2. Evaluation should be multi-dimensional. Beyond the capability ceiling, instruction-following degree, predictability, and behavioral consistency should equally be incorporated into the model evaluation system. The existence of specialized benchmarks like IFEval is precisely a response to this need.
3. The cost-effectiveness of local open-source models continues to improve. For a large number of routine development tasks, 27B-class local models are already an option worth serious consideration. The maturity of quantization technology has turned "running powerful models on your own machine" from a geek experiment into mainstream practice.

Of course, a single experiment isn't enough to draw conclusions. Opus adding wallhacks might just be incidental behavior under a specific prompt. But it's precisely these real observations from the community, delivered with a tongue-in-cheek tone, that often remind us: a so-called "stronger model" isn't always the "better choice" in specific scenarios.

Key Takeaways