Handwritten C/CUDA Inference Engine: Pushing Qwen 35B to the Limits on RTX 5090

Why Build an Inference Engine from Scratch

In the LLM inference space, mature frameworks like PyTorch, vLLM, and llama.cpp already dominate the landscape. Yet a dedicated group of developers continues to build inference engines from the ground up in pure C/CUDA, targeting specific hardware and models for maximum performance. A project that recently appeared on Hacker News is a prime example of this philosophy: a high-performance C/CUDA inference engine built specifically for the Qwen 3.6 35B model on the RTX 5090 (Blackwell architecture).

The value of such projects isn't to replace general-purpose frameworks — it's to explore the boundaries of hardware performance. When you strip away all the abstraction layers and program directly against CUDA kernels and a specific GPU architecture, you can often extract throughput and latency gains that general frameworks simply can't match. For researchers and systems engineers, this is also one of the best ways to develop a deep understanding of the underlying mechanics of modern LLM inference.

RTX 5090 and the Opportunities of Blackwell

The RTX 5090 is built on NVIDIA's latest Blackwell architecture. Released in 2024, Blackwell is NVIDIA's next-generation architecture for consumer and data center markets, succeeding Hopper (the generation that includes the H100). Key innovations include fifth-generation NVLink interconnects, native FP4/FP6 tensor computation support, and an improved Transformer Engine. Compared to the previous Ada Lovelace generation, Blackwell delivers roughly 70% higher memory bandwidth and doubles FP4 throughput over FP8 — a particularly significant advantage for quantized inference, where quantized operations can be hardware-accelerated directly on tensor cores rather than emulated in software.

Why Target a Specific GPU Architecture

General-purpose inference frameworks, by necessity of supporting broad hardware compatibility, often fail to fully exploit the unique capabilities of any single GPU generation. A handwritten CUDA engine can:

Tailor kernel scheduling to Blackwell's tensor core instruction set
Precisely manage memory layouts to reduce data movement overhead between HBM and SMs
Fine-tune for the RTX 5090's specific cache hierarchy and register count
Fully leverage the new architecture's hardware acceleration for low-precision data types

In practice, these optimizations routinely yield double-digit percentage performance improvements — especially during the decode phase, where memory bandwidth is typically the bottleneck.

The Challenge of Running Qwen 35B on a Single GPU

The Qwen series is an open-source large language model family from Alibaba, widely popular in the community for its strong performance in both Chinese and English tasks. At 35B parameters, deploying on a single GPU presents significant memory challenges even with quantization.

Balancing Memory Capacity and Quantization Precision

A 35B model in FP16 requires approximately 70GB of VRAM — far beyond the RTX 5090's 32GB capacity. For single-GPU deployment, quantization is essentially mandatory. Model quantization compresses floating-point weights into lower-precision integer or special floating-point formats. Modern quantization methods like GPTQ and AWQ use layer-wise calibration to significantly mitigate accuracy degradation:

INT4/FP4 quantization: Compresses weights to 4 bits, reducing memory footprint to roughly 18–20GB and leaving sufficient headroom for the KV Cache and activations. Blackwell's native FP4 hardware support further reduces the performance cost of quantization.
Fine-grained KV Cache management: KV Cache grows rapidly in long-context scenarios, requiring proactive memory reuse and eviction strategies.
Operator fusion: Merging LayerNorm, attention, MLP, and other operations into a single kernel reduces launch overhead and eliminates intermediate read/write operations to VRAM.

A handwritten engine has far more freedom in all of these areas, enabling end-to-end joint optimization based on model architecture and hardware characteristics.

Core Technical Highlights of the C/CUDA Inference Engine

The key to building a high-performance inference engine lies in fine-grained control over the execution flow of the computational graph.

Four Major Performance Optimization Dimensions

Based on engineering practice, this type of engine typically focuses on the following areas:

Efficient attention kernels: Borrowing the tiled computation approach from FlashAttention. Proposed by Tri Dao et al. in 2022, FlashAttention's core idea is to decompose attention computation into small tiles via a tiling technique, allowing each tile to reside entirely in the GPU's SRAM (shared memory) and avoid repeated reads and writes to HBM (high-bandwidth memory). This reduces the memory complexity of attention from O(N²) to approximately O(N), cutting bandwidth pressure by 5–10x in long-sequence scenarios — a critical optimization.
Custom GEMM kernels: Matrix multiplication is the core computational bottleneck of Transformers. Handwriting or deeply tuning GEMM kernels for Blackwell tensor cores can significantly boost overall throughput.
Prefill/Decode pipeline scheduling: LLM inference consists of two fundamentally different computation phases — the Prefill phase processes all input tokens at once and is compute-bound; the Decode phase autoregressively generates one token at a time but must load the entire KV Cache for each step, making it memory-bound. The two phases place very different demands on GPU resources, and applying differentiated scheduling strategies (such as continuous batching) can maximize GPU utilization and minimize idle time.
Pre-allocated memory pools: Pre-allocating a fixed memory pool before execution eliminates fragmentation and latency jitter caused by frequent runtime allocations and deallocations.

The Real-World Trade-off: Performance vs. Maintainability

The costs of handwriting in pure C/CUDA are considerable — development complexity increases dramatically, and code maintainability and portability suffer significantly. Every new model architecture or GPU generation may require substantial rewrites. This is precisely why such projects tend to focus on a specific "model + hardware" combination: pursuing peak performance in a vertical use case rather than generality.

Implications for Developers and the Industry

This project reflects a trend worth watching closely: as large models move into production deployment, inference performance optimization is becoming a core competitive battleground.

For individual developers and small teams, open-source implementations like this are excellent learning resources, offering deep insight into the underlying mechanics of LLM inference — from quantization principles and attention kernel design to Prefill/Decode scheduling strategies, each area is worth deep exploration. For enterprises deploying in cost-sensitive inference scenarios, deep customization targeting their own hardware stack can yield substantial efficiency gains and cost savings.

As Blackwell and subsequent GPU architectures become more widely adopted, we can expect a growing wave of high-performance inference solutions targeting new hardware. The deep co-optimization of hardware and software will continue to push the price-performance frontier of large model inference ever higher.