Why Modernization Breaks Working Code: An Empirical Study of LLMs on Legacy PHP

On the Hacker News tech community, a discussion titled "When Modernization Breaks Working Code: A Study of LLMs on Legacy PHP" has drawn attention. The study focuses on an overlooked yet pervasive real-world problem: when developers try to use large language models (LLMs) to "modernize" legacy PHP code that is still running in production, are the model's suggested changes actually safe and reliable?

Why Legacy PHP Is a Special Testbed

Legacy PHP codebases typically have two characteristics: first, their syntax and design patterns are stuck at earlier versions; second, after years of patching, they have accumulated a large number of "runs-but-ugly" implicit dependencies. Such code often lacks complete test coverage, and documentation is severely missing. LLMs excel at completing code based on common patterns, but when they face non-standard implementations laden with historical baggage, their tendency to "rationalize" may instead introduce risk.

This is precisely the starting point of the study: modern development tools default to pursuing conciseness and modern syntax, while what legacy systems need most is behavioral consistency. The tension between the two constitutes the core challenge of LLM-assisted refactoring.

Implicit dependencies and behavioral contracts: A common "implicit dependency" in legacy systems means that the correctness of the code depends on external conditions not explicitly declared in interfaces or documentation, such as the initialization order of global variables, the timing of function calls, the shared state of database connections, or the way include paths are resolved under a specific PHP version. These dependencies are often inadvertently formed over years of patching and have never been formally recorded. A behavioral contract refers to the set of observable behaviors a system exhibits externally, including return values, exception types, side effects, and performance boundaries. For legacy systems, the real goal of refactoring is not to make the code "clean" but to keep the behavioral contract unchanged. LLMs cannot perceive these implicit constraints when generating code, so even if their output is syntactically correct, it may silently alter the contract at boundary conditions or error paths.

Core Phenomena Observed in the Study

According to the study's title and discussion content, a recurring pattern emerges when LLMs modify legacy PHP: the model actively replaces constructs that look outdated but actually carry critical business logic. For example, it rewrites code that depends on a specific execution order into a more "elegant" structure, or replaces an old-style database access layer with calls to a modern framework. These changes are often syntactically correct, yet they may alter boundary conditions, error-handling paths, or the scope of influence of global state.

This is not a matter of insufficient model capability, but rather a misalignment between its objective function and the maintenance goals of legacy systems. LLMs are trained to generate "typical" good code, whereas what legacy systems need is precisely conservative modification that "keeps things as they are."

Objective function misalignment: In machine learning, the objective function defines the direction the model optimizes during training. For code-generation LLMs, training data comes largely from "good code" in open-source projects that is stylistically consistent and clearly structured, so the model learns to favor common, concise, modern formulations. But the maintenance goals of legacy systems are precisely the opposite: prioritize behavioral equivalence and minimal changes over code aesthetics. This misalignment means the model will systematically "correct" constructs that appear outdated yet carry critical logic, such as rearranging statements that depend on execution order into a more functional structure, or replacing old-style mysql_* calls with the PDO abstraction layer. These changes may not be semantically equivalent, because old-style code may depend on specific error suppression, resource-release timing, or global state. Understanding this misalignment helps developers anticipate the risk points in model output rather than simply blaming insufficient model capability.

Practical Implications for Developers

The signal from this study is clear: when using LLMs for modernization in legacy codebases, model output cannot be applied directly as a patch. A safer approach is to position the LLM as an aid for understanding code rather than an executor of automatic refactoring. Before applying any model-generated change, targeted regression tests must be relied upon to verify behavioral equivalence.

For legacy projects lacking tests, this means writing tests first before discussing refactoring. LLMs can accelerate the generation of test cases, but manually confirming whether those cases cover critical paths remains an indispensable step.

Regression testing and behavioral equivalence: Regression tests are a set of test cases run repeatedly before and after a change to confirm that existing functionality has not been broken. In the context of LLM-assisted refactoring, the core role of regression tests is to verify "behavioral equivalence"—that the modified code produces exactly the same outputs and side effects as before for all covered inputs and states. For legacy PHP projects, this usually means first supplementing critical paths with characterization tests, i.e., tests that record the system's actual current behavior rather than tests based on specifications. LLMs can assist in generating the skeleton of these tests, but developers must manually review whether the cases cover exception branches, global state changes, and the timing of database interactions. Without this layer of verification, any model-generated "modernization" patch is merely a syntactically correct guess.

Conclusion

Although brief, this discussion on Hacker News touches on a problem in AI-assisted programming that is easily obscured by hype: code being "modern" and code being "correct" are not always the same thing. In legacy system scenarios, the criterion for measuring LLM output quality should not be how new the code is, but whether it respects the system's existing behavioral contract.

One-line Summary

Study reveals that LLMs modernizing legacy PHP tend to replace old constructs carrying critical logic, potentially breaking behavioral consistency.

Paragraph Summary

This study focuses on the safety of using large language models to modernize legacy PHP code. Legacy PHP codebases have outdated syntax, many implicit dependencies, and low test coverage, while LLMs tend to generate "typical" modern good code and actively replace constructs that look outdated but carry critical business logic, thereby altering boundary conditions, error-handling paths, or the scope of global state influence. This is not insufficient model capability but a misalignment between its objective function and legacy system maintenance goals. The study recommends positioning LLMs as aids for understanding code rather than executors of automatic refactoring; any model-generated change must be validated for behavioral equivalence through regression tests; projects lacking tests should add tests first, with LLMs able to assist in generating cases but humans needing to confirm coverage of critical paths.