Closing the Binary Blind Spot: Domain-Knowledge Injection Improves LLM Detection of Malicious npm Packages

Supply Chain Attacks Are Escalating — So Why Do LLMs "Miss" on npm

Supply chain attacks on package registries are evolving rapidly in both volume and technique. The npm ecosystem is especially exposed — a single poisoned package can be pulled into tens of thousands of projects within hours. Traditional static analysis tools excel at matching known signatures, but they struggle with obfuscation, dynamic loading, and binary payloads.

Large language models have been widely expected to "read" package behavior the way a security analyst would. The reality, however, is that LLMs have a clear binary blind spot when triaging malicious npm packages: once malicious logic is hidden inside compiled .node files, WASM, or obfuscated bytecode, the model only sees a pile of semantically meaningless printable characters, and its judgment degrades sharply.

The study titled Closing the Binary Blind Spot targets exactly this gap, proposing domain-knowledge injection — feeding the model the structured priors required for security analysis so that it is no longer "blind" when facing binary payloads.

Where Exactly Does the Binary Blind Spot Come From

LLMs Can Read Source Code, But Not Binaries

npm packages are not always pure JavaScript. Native extensions (.node), WebAssembly modules, and minified mega-bundles are all common forms. LLM pretraining corpora are dominated by natural language and readable code, so their modeling capacity for binary byte streams is inherently weak.

Researchers point out that malicious authors exploit precisely this: they embed the core payload in binary files and leave the model staring at a harmless JS wrapper. The "benign" verdict the model produces is, in fact, built on incomplete information.

Native extensions (.node) are Node.js ABI-compatible binary modules, typically compiled from C/C++ via node-gyp and loaded at runtime as dynamic libraries. They can directly invoke system calls and low-level APIs, and can therefore execute arbitrary machine code, bypassing the JavaScript sandbox and static analysis. WebAssembly (WASM) is a portable binary instruction format that, although it runs inside a restricted sandbox, also exists as bytecode whose true behavior is hard to reconstruct through ordinary text-level scanning. What they share is this: source-level semantics are largely lost after compilation, the string table retains only a few readable symbols, and malicious intent is hidden within the instruction and control flows. This is the direct cause of the "binary blind spot" LLMs exhibit on npm packages — what the model sees is a semantically meaningless byte sequence, not the code structure it is familiar with from pretraining.

The Tension Between Context Window and Signal-to-Noise Ratio

Another practical constraint is context length. A mid-sized npm package can easily exceed hundreds of thousands of lines once decompressed, and stuffing all of it into the prompt is both expensive and dilutes the signal. If you convert binaries into hex and feed that to the model, token consumption explodes even further, and the signal-to-noise ratio becomes worse.

This means simply "letting the model see more" is not the answer. The key is letting the model see the right things — which is exactly the problem domain-knowledge injection aims to solve.

Domain-Knowledge Injection: Handing the Model a Security Expert's Judgment Framework

What's Injected Is Not Data, But Structure

The core idea of the paper is not to pile up samples, but to inject structured knowledge from the security analysis domain into the triage pipeline. This includes: common attack patterns in malicious packages, typical characteristics of binary payloads, associations between package metadata and code behavior, and so on.

This knowledge is organized in a form the model can digest — essentially handing the model an "analyst's checklist" before inference, guiding it to focus on high-risk signals instead of scanning everything with equal effort.

Domain-Knowledge Injection is a method that explicitly encodes expert priors into the inference chain, as distinct from fine-tuning or retrieval-augmented generation (RAG). Fine-tuning changes model weights, which is costly and slow to update; RAG supplements external text at inference time, but the input remains unstructured fragments. Knowledge injection emphasizes organizing analytical frameworks, judgment rules, and risk signals into structured prompts the model can digest (such as checklists, rule templates, and feature comparison tables), placed up front before inference. This way the model doesn't have to "intuit" patterns from massive raw data on its own, but rather follows a path set by experts to focus on high-value information. In malicious package detection, such knowledge typically includes: common poisoning techniques, suspicious install-script behaviors, network callback patterns, obfuscation and packing characteristics, and more.

Making Binary Features "Interpretable"

For the binary portion, the study combines feature extraction with semantic mapping: the import table, section characteristics, suspicious API calls, and so on are distilled into structured descriptions, then handed to the model for reasoning. The model is still processing textual semantics — what it is good at — but the input already contains the key binary information.

According to the study's description, this treatment yields marked improvement in both accuracy and consistency on samples involving binary payloads — the blind spot is partially filled rather than bypassed.

Implications for Security Engineering Practice

A Single Model Is Not a Silver Bullet

One sobering signal from the research: wiring an LLM directly onto npm packages to render a "yes/no malicious" verdict will systematically underestimate threats containing binary payloads. Any automated triage system that goes into production needs an explicit binary-handling stage.

Knowledge Injection Is a Reusable Engineering Paradigm

The value of domain-knowledge injection extends beyond malicious package detection. Wherever a large model faces a specialized vertical domain and the raw data falls outside its pretraining distribution, this paradigm is worth borrowing — put expert knowledge up front in the inference chain rather than expecting the model to "figure it out" on its own.

Evaluation Must Cover Binary Samples

If a benchmark set is almost entirely pure-JS malicious packages, the model's performance will be overestimated. When building security-focused LLM evaluation benchmarks, binary payload samples should be a mandatory component; otherwise, what is measured is merely a "simple mode" score.

The sample composition of a benchmark directly determines the credibility of its conclusions. If a security-focused LLM evaluation set is dominated by pure-JavaScript malicious packages, a model can easily score high by matching obfuscated strings, suspicious domains, or install scripts — but that does not reflect real-world attack-defense capability. Binary payload samples are inherently harder to detect: they do not expose readable source code, and their behavior must be inferred indirectly through import tables, section entropy values, API call sequences, and so on. Without such samples, evaluations will systematically overestimate a model's performance in scenarios involving native extensions, WASM, or packed payloads. Therefore, when building security LLM benchmarks, samples should be stratified according to the real proportion of binary packages in the npm ecosystem, and accuracy and recall on the binary subset should be reported separately.

Questions Still to Be Answered

This study fills part of the blind spot, but the boundary remains fuzzy. Binary feature extraction itself may be evaded by adversarial construction; how the injected knowledge evolves continuously with attack techniques is also a long-term challenge. Moreover, whether inference costs in low-resource scenarios become a deployment bottleneck has not yet been adequately discussed in the original work.

What is certain is that the combination of supply chain security and large models will not stop at "throw the code at the model." Whoever can better engineer domain knowledge into the model will be more likely to seize the initiative in real-world attack and defense.

Background Notes

Import Table records the external functions and dynamic libraries that a binary module depends on, and is key to determining its capability boundary. For example, importing Windows APIs such as CreateRemoteThread and VirtualAllocEx often points to process injection, while importing socket and connect suggests network communication. Section Characteristics describe the permissions and entropy values of PE/ELF sections: writable and executable sections (W+X) are a strong signal of self-modifying code or packing, and high-entropy sections often mean encrypted or compressed payloads. Suspicious API calls include file writes, registry modifications, process creation, anti-debugging checks, and so on. Once these raw binary attributes are distilled into structured textual descriptions, the model can reason about behavioral intent at the semantic level without needing to understand the bytecode itself — thereby converting unreadable binaries into judgeable threat signals.

One-line Summary

This study proposes domain-knowledge injection to close LLMs' binary blind spot in malicious npm package detection.

Paragraph Summary

This study focuses on the "binary blind spot" of large models in npm supply chain security triage: when malicious logic is hidden inside .node native extensions, WASM, or obfuscated bytecode, LLMs' judgment degrades sharply due to a lack of binary modeling capability, and they are often misled by harmless JS wrapper layers. The paper proposes a domain-knowledge injection paradigm that places structured priors from security analysis (attack patterns, binary features, metadata association rules, etc.) in a model-digestible form up front in the inference chain, and through structured descriptions such as import tables, section characteristics, and suspicious API calls, enables the model to process key binary information at the semantic level, thereby partially filling the blind spot. The article further notes that automated triage systems must explicitly include a binary-handling stage, and security evaluation benchmarks must cover binary samples; otherwise, model performance will be systematically overestimated.