LuaJIT Performance Trap: How NYI Silently Poisons Unrelated Hot Loops

A Bizarre Performance Mystery

While optimizing a Lua transpiler for the modding language grug, a developer encountered an extremely subtle performance issue in LuaJIT: the same benchmark would inexplicably run 20x slower at times. This kind of non-deterministic performance jitter is among the hardest class of bugs to diagnose—it's not a logic error, it doesn't throw an exception, it just silently tanks your runtime speed.

After deep investigation, the root cause was traced to one of LuaJIT's NYI (Not Yet Implemented) features. What made it particularly insidious was that this NYI could "silently" poison a hot loop that had absolutely no logical connection to it, causing an overall performance collapse. This case reveals an easily overlooked dark corner in JIT compiler internals.

LuaJIT's Trace Compilation Mechanism Explained

To understand this bug, you first need to understand LuaJIT's core operating principle. LuaJIT uses trace-based just-in-time compilation (trace-based JIT), which is fundamentally different from traditional method-based JIT.

Traditional method-based JIT compilers (like HotSpot JVM's C2 compiler) use entire methods or functions as compilation units, compiling a complete function into machine code. Trace-based JITs (like LuaJIT, or the early Mozilla TraceMonkey) use actual execution paths as compilation units—they only compile the linear path the program actually traverses, recording only the direction actually taken at branches. The advantage: compiled code has no redundancy from untaken branches, enabling more aggressive inlining and optimization. The disadvantage: when the program takes a different path, it needs to "exit" from the compiled trace (side exit), falling back to the interpreter or jumping to another trace. This architecture means the linking relationships between traces are critically important—and this is the structural root cause of the pollution problem described in this article.

Traces and Hot Loops

LuaJIT's trace recorder monitors code execution. When a loop is repeatedly executed and reaches the "hot" threshold, the compiler begins recording every bytecode operation along that execution path, compiling them into highly optimized machine code. This recorded and compiled path is a trace. When execution reaches that loop again, it can directly run the compiled native code, yielding enormous performance gains.

Specifically, LuaJIT uses counters to determine whether code is "hot." By default, a loop becomes hot after 56 executions, and a function becomes hot after 56 calls (these thresholds are adjustable via the -Ohotloop and -Ohotcall parameters). When a counter reaches the threshold, the trace recorder activates and begins recording bytecode execution instruction by instruction. After recording completes, the trace passes through an optimization pipeline—including constant folding, dead code elimination, register allocation, and more—to generate x86/x64 or ARM machine code. The compiled trace is stored in a trace cache, and the next time execution reaches the same entry point, it jumps directly to the machine code.

The Role and Impact of NYI Operations

However, not all bytecode operations can be trace-compiled. Some operations are marked as NYI—meaning LuaJIT's compiler doesn't yet support native code generation for them. When the trace recorder encounters an NYI operation during recording, it cannot continue compilation and must abort the current trace recording.

This itself is by design. The problem lies in what happens after the abort—LuaJIT, to avoid repeatedly attempting to compile paths that are destined to fail, will "blacklist" the relevant code, preventing further JIT compilation attempts. Specifically, LuaJIT maintains an internal penalty mechanism: if the same starting point repeatedly triggers aborts, that starting point's hotness counter is set to an extremely large value (i.e., "blacklisted"), making it virtually impossible to reach the compilation threshold for the remainder of the program's lifetime. The design intent is reasonable—avoiding wasted CPU cycles on paths that will inevitably fail. But the critical hidden danger is that a trace's starting point is not necessarily where the NYI operation resides, and this lays the groundwork for cross-boundary pollution.

How NYI Pollution Crosses Code Boundaries

What's truly surprising about this case is that an abort and blacklisting caused by an NYI operation ended up affecting a hot loop that was completely unrelated to it logically.

From Performance Mystery to Internal Mechanisms

Starting from the benchmark anomaly, the developer dug all the way down to LuaJIT's trace recorder implementation. Trace compilation doesn't operate in isolation on individual loops—traces can have linking and nesting relationships with each other. When a trace aborts due to encountering an NYI operation like unpack, the abort and subsequent blacklisting mechanism can, under certain conditions, implicate nearby or associated code paths.

The mechanism behind this "guilt by association" works as follows: a trace might begin recording from loop A, and during recording, due to inlining or other reasons, enter function B, where B contains an NYI operation that causes an abort. In this case, what gets blacklisted is loop A's starting point, not function B itself. Loop A might have zero logical connection to the NYI operation at the source code level, but from the trace recorder's perspective, they exist on the same trace's recording path.

The result: a hot loop that should have been smoothly JIT-compiled and running at high speed gets stripped of its compilation eligibility because of an NYI operation that has nothing to do with it, and is forced to fall back to interpreted execution mode. The 20x performance gap is precisely the chasm between "compiled execution" and "interpreted execution." This also explains why the performance behavior was random—it depended on the exact timing of trace recorder aborts and blacklisting, which is influenced by multiple dynamic factors including code execution order and counter states.

The NYI Problem with unpack

The investigation ultimately pointed the finger at unpack, a commonly used function. At the time, it was on LuaJIT's NYI list, meaning traces involving it could not be fully compiled.

unpack (renamed to table.unpack in Lua 5.2+) is a core built-in function in Lua used to expand table elements into multiple return values. Typical usage: local a, b, c = unpack({1, 2, 3}). In practice, unpack is widely used for variadic argument passing, argument expansion in function calls, and multi-value destructuring after pattern matching. For transpilers, unpack is even more critical—when the source language has tuple destructuring, multiple return values, or argument spreading, transpiling to Lua naturally generates extensive unpack calls. LuaJIT's NYI list is documented on its official wiki; other common NYI operations include the next iterator for pairs, the iterator for string.gmatch, and certain metamethod operations.

For transpiler output code that heavily uses unpack, this is virtually an invisible performance killer.

From Investigation to Upstream Fix

The value of this case lies not just in discovering the problem, but in the developer following the trail to deliver a definitive fix: submitting a PR to remove unpack from LuaJIT's NYI list, allowing it to be properly trace-compiled.

This is a textbook example of open-source community collaboration—a downstream project (the grug language's Lua transpiler) contributed an improvement back to upstream infrastructure (LuaJIT) during its optimization work. Once unpack supports JIT compilation, it not only eliminates this specific pollution problem but universally benefits all LuaJIT code that relies on unpack.

Practical Takeaways for Developers

This investigation process offers several important insights for developers working with JIT-compiled languages:

Be vigilant about NYI operations in LuaJIT. In LuaJIT, seemingly ordinary built-in functions can break trace compilation because they're on the NYI list. Consulting LuaJIT's NYI documentation and avoiding these operations on hot paths is fundamental to performance optimization.

Non-deterministic performance issues often point to JIT mechanics themselves. When the same code exhibits massive performance swings, it's likely related to trace aborts, blacklisting, and compilation timing rather than business logic. Use LuaJIT's diagnostic tools like -jv, -jdump to observe trace recording and abort behavior. Specifically, -jv (verbose mode) prints compilation and abort information for each trace, including the reason and location of aborts; -jdump outputs the trace's IR (Intermediate Representation) and final generated machine code, suitable for deep analysis; -jp is the built-in profiler that shows whether time is being spent in JIT-compiled code or interpreted code. When investigating the pollution problem described in this article, -jv output will show messages like TRACE 5 abort: NYI: unsupported builtin, which is the first clue for locating the issue.

Performance pollution can cross code boundaries. Don't assume that a function's NYI problem only affects itself. The JIT compiler's traces form an interconnected whole, and one abort can implicate seemingly unrelated hot loops.

Transpilers need deep adaptation to target runtimes. When a language chooses to transpile to another language (e.g., TypeScript→JavaScript, grug→Lua), the code patterns generated by the transpiler profoundly affect the target runtime's optimization effectiveness. This is known as "target-friendly codegen." For example, V8 is sensitive to hidden classes, so TypeScript transpilers need to ensure consistent object property initialization order; similarly, LuaJIT is sensitive to trace continuity, so transpilers should avoid generating NYI operations in hot paths. This requires transpiler developers to understand not only the source language's semantics but also the target runtime's JIT behavioral characteristics, choosing the most JIT-friendly option among semantically equivalent code generation approaches.

For projects like grug that transpile a high-level language to Lua, the generated code patterns directly determine whether LuaJIT can compile efficiently. Understanding the target runtime's JIT behavior is a required course for writing high-performance transpilers. This investigation began with a 20x performance mystery and concluded with an upstream PR—a textbook example of performance issue root-cause analysis.

Key Takeaways