Reverse-Engineering a CUDA Inference Engine with Ghidra: 12% Speedup on an RTX 3090

Reverse engineering is usually seen as a tool for security research or cracking, but one developer's practice shows it can also be used for performance tuning of local AI inference engines. This user runs a never-shut-down coding model service on an RTX 3090, uses Ghidra to read the engine's symbol table, and combines it with real-world session access patterns to ultimately achieve a 12% speedup on the decode side and roughly a 25x reduction in session-switching latency.

Read the binary first, then change the parameters

Before adjusting any configuration, the developer performed static analysis on the inference engine Strata 0.1.44 using Ghidra 12.1.4. The engine runs under a systemd service, listens on port 8090, is proxied by llama-swap on port 8080, and shares a GGUF model collection behind it.

Binary analysis revealed a lot of key information. The .text section has 8.88 MiB of host code, and the CUDA fatbin image totals 50.3 MiB, all compiled for the sm_86 architecture—exactly matching the RTX 3090, so there are no JIT compilation surprises at startup. The symbol table is not stripped, containing 12,446 procedures and 54,524 strings, and is an NDEBUG build with -O3 optimizations.

The symbol names confirm the architectural design: core::Verifier::run works with AdaptGate to verify during expert exchange windows, RemoteExpertOpt::adapt and PeerExperts::adapt implement the adaptive expert layer, detect_cpu_topology_impl and PoolAffinity support CPU pinning for 7 worker threads, and prefill::detail::mmq_resident_sort_rows is the prefill path. The documentation, the symbol table, and the real-time stage timings from STRATA_DECODE_TIMING corroborate this architecture from three angles.

Startup logs show: the expert cache has 7325 slots occupying 13.94 GiB of VRAM (out of 24.5 GiB total), the KV cache has 32768 rows resident plus 6.19 GiB pinned in memory, and the hit rate is stable at 94-96.7%. The MTP draft head is 212.9 MiB, --spec-min-p is auto-tuned from 0.55 to 0.35, and --pcie-frac is fixed at 0.20. VRAM usage is 23.8/24.5 GiB, which means any optimization must fit into the remaining space.

About Ghidra and static reverse engineering

Ghidra is a software reverse-engineering framework open-sourced by the U.S. National Security Agency (NSA), supporting disassembly, decompilation, and symbol analysis, and can restore compiled binaries into approximately readable C code. In this scenario, the developer did not use it to crack software, but rather treated the inference engine's binary as a "map" to read: the unstripped symbol table (containing 12,446 procedures and 54,524 strings) is effectively a natural architecture document, and the function names directly expose module responsibilities, such as core::Verifier::run, RemoteExpertOpt::adapt, prefill::detail::mmq_resident_sort_rows, and so on.

It is worth noting that the CUDA fatbin is a container format used by NVIDIA's compiler to embed device-side GPU code into a host binary, containing SASS instructions precompiled for a specific compute capability (such as sm_86, corresponding to the Ampere architecture of the RTX 3090). Confirming that the fatbin is entirely compiled for sm_86 means the engine will not trigger just-in-time compilation (JIT) at startup, thereby ruling out performance jitter caused by JIT warm-up and making subsequent measurements more credible. -O3 NDEBUG indicates that this is a production build with heavy optimization and assertions disabled, with performance already close to the upper limit the compiler can provide; the tuning space is mainly in runtime parameters rather than the code itself.

The most important failure: cross-session prefix caching

The developer's actual usage pattern is not a single conversation, but multiple sessions alternating—switching back and forth between scheduled tasks, group chats, calls, and long conversations. Therefore, the probe he designed accurately simulates this scenario: two conversations alternate, and the latency of returning to the first conversation is measured.

The baseline results exposed a serious problem. Returning to the first conversation required a full re-read of 10.4 to 10.8 seconds. Each switch discarded the content that session had already read. Within a single session, prefix caching was already effective—repeating a 30k prompt in the same session took only 0.07 seconds, and appending turns took 0.07 to 0.32 seconds. But nothing was retained across sessions, and each switch paid the full re-read cost.

The solution was three parameters: --prompt-cache 3 --conversation-cache-mib 4096 --conversation-cache-slots 4. Once enabled, parked conversations were retained across alternating switches, and the same test dropped from 10.8 seconds to 0.4 seconds. An unexpected side effect was that a shared prefix across sessions hit the same system prompt root checkpoint, so scheduled tasks returned at almost zero cost from a cold start, and the 30k read after parktest dropped from 10.3 seconds to 5.3 seconds.

But the cost came with it. Under the 4096 MiB and 4-slot configuration, small in-session reads increased from a clean 0.07 seconds to an extra 0.15 to 0.35 seconds per turn. The agent loop is precisely a continuous small-read scenario. After dropping to 3072 MiB and 3 slots, occasional tail latency returned to near the original level. The final configuration was smaller than the first one that worked—because the cost was the reason.

About prefix caching and KV caching

The key to understanding this section is distinguishing two layers: KV cache and prefix cache. The KV cache (key-value cache) is the core mechanism of Transformer inference—for each token generated, the model needs the attention key-value pairs of all previous tokens; caching them avoids recomputation, but memory usage grows linearly with context length. In the text, 32768 rows of KV cache are resident and 6.19 GiB is pinned in memory, precisely for this reason.

Prefix caching operates at a higher level: when the beginning of a new request is exactly the same as a previous request, the engine can directly reuse the KV state corresponding to that prefix and skip the entire prefill. But traditional implementations usually only retain the prefix of the "most recent" request; once you switch to another session, the original prefix is overwritten. The developer's --conversation-cache-slots 4 essentially lets the engine keep independent prefix snapshots for multiple sessions (parked conversations), which is exactly why cross-session switching dropped from 10.8 seconds to 0.4 seconds.

The cost is VRAM: each slot occupies a copy of the KV state, and 24.5 GiB of VRAM is already divided up among the expert cache (13.94 GiB), KV residency, the MTP draft head, and so on, so the number of slots and MiB must be carefully balanced—only after dropping to 3072 MiB/3 slots did the extra overhead of small in-session reads return to an acceptable range.

The trap of greedy sampling

The second failure came from request shape. The developer's agent stack sends requests to the local provider without sampling parameters, and the engine uses default greedy decoding when no sampling parameters are received. Greedy mode produces loops on the IQ1_M quantized coding model—the same token sequence repeats over and over.

The fix was to add a sampling configuration block to the service JSON: temperature 0.6, top_p 0.95, top_k 20. Lines 20-24 of runconfig.py confirm that these keys are correctly received. Decoding speed remained within the probe range; what was gained here was correctness, not performance.

About greedy decoding and IQ1_M quantization

Greedy decoding means selecting the highest-probability token at each step without introducing any randomness. It seems "most accurate," but on low-bit quantized models it is extremely prone to degenerate loops: because weight quantization loses a large amount of numerical precision, the model's scores for certain token sequences develop pathological high peaks, causing the same passage to be output repeatedly and unable to break out on its own.

temperature, top_p, and top_k are exactly the three knobs that break this determinism. temperature 0.6 divides the logits by the temperature before applying softmax, making the distribution flatter; top_p 0.95 (nucleus sampling) keeps only the candidates within the top 95% cumulative probability; top_k 20 further limits the number of candidates. Together, these three introduce moderate randomness into sampling, thereby bypassing the high-score trap of quantized models.

IQ1_M is an extreme quantization format at roughly the 1-bit level in the llama.cpp family (the "M" indicates mixed precision), designed specifically to fit large-parameter models into limited VRAM, at the cost of reduced output quality and the aforementioned loop risk. This also explains why the developer emphasized that this fix "gained correctness, not performance"—decoding speed did not change, it just made the model stop repeating itself.

Overlap optimization of expert exchange windows

A measurement of one decode window showed a total time of 18.76 milliseconds, of which routing plus planning plus bell took 2.36 milliseconds, VRAM hit took 2.29 milliseconds, waiting took about 1.1 milliseconds, and the head took 0.93 milliseconds. The expert layer swaps slots between windows, which means the swap in each window is pure waiting time.

After enabling STRATA_ADAPT_OVERLAP=1, the swap is completed during decoding. The changed log shows "across 130 windows the swap was still landing, with total waiting of 0.11 milliseconds," window time dropped from 18.76 milliseconds to 17.73 milliseconds, and the decode median improved from 91.9-92.6 to 95.6. Another option, --adapt-async 1, had a median of 92.6; the two conflicted, so the overlap approach won and the async approach was abandoned that same day.

About expert exchange and computation overlap

The engine in the text uses an adaptive expert layer (RemoteExpertOpt::adapt, PeerExperts::adapt) architecture, analogous to the MoE (mixture of experts) idea: different inputs activate different subsets of experts. When the required expert is not in VRAM, slots must be swapped between decode windows from host memory or a peer. In a traditional serial flow, this swap happens after one window ends and before the next begins, so the GPU can only wait idle, measured at about 1.1 milliseconds of pure waiting.

The role of STRATA_ADAPT_OVERLAP=1 is to move the swap operation to occur concurrently with decode computation (that is, using CUDA streams or async copies concurrently with compute kernels), allowing PCIe transfers and GPU compute to mask each other. The log line "across 130 windows the swap was still landing, with total waiting of 0.11 milliseconds" is evidence that overlap is working—the fixed wait that originally had to be paid every window was compressed to nearly zero.

The author also compared --adapt-async 1, whose decode median was 92.6 versus 95.6 for the overlap approach, indicating that although the async approach also attempts parallelization, its scheduling granularity or synchronization points differ and instead introduce additional overhead, so it was abandoned. This case shows that the benefits of concurrency optimization depend heavily on the specific implementation path and must be chosen using measured data rather than intuition.

Two engineering traps

serve/server.py starts the engine with cached parameters, so editing the JSON file has no effect until the service is restarted. The file on disk looks correct, but the running engine's parameters are different. A restart takes 18-19 seconds, after which polling /v1/models and performing a real conversation probe are required to confirm.

The calibrate command starts its own engine instance and monopolizes VRAM, so the service must be stopped first. When the developer first ran calibration, the worker threads and the adaptive expert layer branches all failed to start because the just-stopped engine still held the GPU. Every number from that run was invalid.

Options tested and then abandoned

The developer also tested six seemingly reasonable parameters that had no effect, each with recorded data:

• KV prefetch: activated successfully and recorded layer prefixes; 18.0 seconds both before and after 66k prefill. Completely ineffective on PCIe 4.
• Fused int8 prefill kernel: activated successfully; 30k prefill went from 10.44 seconds to 10.53 seconds. Useless on IQ1_M quantization.
• --prefill-head-alias: requires disabling short reads, and every prompt read pays a 0.3-0.6 second head reload cost. The agent loop is a continuous small-read scenario, so this is net negative; skipped after analysis.
• Huge pages: the engine needs 11,992 mappings of 2 MiB, but the kernel returns only 11,170, fragmentation is severe, MAP_HUGETLB is always unavailable, and reservation would waste 22G of memory for nothing. Rolled back to zero.
• Rejection-guided speed prediction: 0.2-0.4% cost per token, but it changes model behavior. Abandoned.
• Upstream version 0.1.44.1: 11 commits of fixes, no performance-related changes. Skipped to avoid risk.

Final data and lessons learned

Under the same probe, cross-execution, and same-session conditions, short decode improved from 108.7 tok/s to 121.5 tok/s, a 12% gain. Prefill was essentially flat from 2998 tok/s to 3004 tok/s—all the gains were on the decode side. Session alternation dropped from 10.4-10.8 seconds to 0.2-0.5 seconds, roughly a 25x improvement. Persistence mode (nvidia-smi -pm 1) was also enabled; that setting was originally off and does not persist across restarts.

The developer gave two core lessons. First, measure access patterns before tuning parameters—the 25x improvement came from the way sessions alternate and had nothing to do with the engine itself; a probe that always kept a single conversation would never have discovered this problem. Second, dead ends are themselves deliverables—the six seemingly viable parameters each came with data, so the next person does not have to spend another day retesting them.

The hardware configuration was a Ryzen 7 5800X, 64G DDR4, and an RTX 3090 24G, with measured host-to-device bandwidth of 25.9 GB/s on PCIe 4.0. Testing used the IQ1_M coding quantized model and a cold-cache probe.