qwenGPU: Compiling a SIMT GPU Into the 45 Weight Layers of Qwen3.5-2B

Compiling a 4-lane SIMT GPU into the weights of a language model sounds like a rather transgressive use of the Transformer architecture: the qwenGPU project compiles the GPU's parallel execution logic directly into Qwen3.5-2B, forming 45 untrained parameter layers and letting the model itself write out every step of a real CUDA launch. According to MarkTechPost, the heart of this experiment is using the Gated DeltaNet state to serve simultaneously as registers and memory.

What's inside the 45 untrained weight layers

qwenGPU does not add a "call CUDA" tool interface to the model; instead, it turns the GPU's hardware behavior itself into the model's internal structure. The instruction stream, thread scheduling, and data paths of a 4-lane SIMT GPU are mapped into 45 layers of weights, which are untrained in their initial state yet carry the execution semantics of the GPU.

This is worth noting: traditionally, having a language model "generate code" and having a language model "simulate hardware" are two entirely different paths. The former relies on the model's memory and compositional ability over APIs, while the latter requires the model to reproduce the evolution of a state machine within its parameter space. qwenGPU belongs to the latter—it treats the GPU as a deterministic system that can be compressed into a neural network.

SIMT stands for Single Instruction, Multiple Threads and is the core of NVIDIA's GPU execution model: multiple threads (usually in groups of 32, called a warp) execute the same instruction but each operates on different data. The 4-lane SIMT GPU mentioned here means only 4 parallel execution lanes per instruction cycle—far smaller than a real GPU, making it easier to map completely into the layer structure of a neural network. SIMT differs from the CPU's SIMD (Single Instruction, Multiple Data) in that it emphasizes thread-level parallelism rather than mere vector width, and each thread has its own registers and program counter. Understanding this is key to understanding qwenGPU: encoding SIMT's execution semantics into weights means the model needs to implicitly represent mechanisms such as thread scheduling, branch divergence, and masking.

Qwen3.5-2B is a lightweight model in the Tongyi Qianwen series, with about 2 billion parameters, typically used for on-device or resource-constrained scenarios. Choosing this scale for a hardware simulation experiment is partly because the small parameter count makes it easy to control variables, and partly because it shows the experiment is not pursuing performance but validating architectural feasibility. The 45 untrained weight layers mean these parameters have not undergone gradient descent to fit real GPU behavior, so they currently carry a human-designed structural prior rather than execution logic learned from data.

Gated DeltaNet state as both registers and memory

The most technically sophisticated design of this experiment is using the Gated DeltaNet state to play the role of both registers and memory. In a real GPU, registers are the fastest storage for execution units, while memory is a tier with larger capacity but higher latency; the state update mechanism of Gated DeltaNet naturally carries the semantics of "read/write" and "retain," which happen to correspond to these two types of storage behavior.

In other words, the model's forward pass is no longer just feature transformation but a simulation of the execution trace of a GPU kernel. Each layer corresponds to a stage in the GPU execution flow, and state passes between layers just as data flows between registers and memory. This mapping gives "model inference" and "hardware execution" a formally isomorphic relationship.

Gated DeltaNet is a class of linear attention/state space model structures with gating mechanisms, whose core is a state matrix that is recursively updated along the sequence dimension. Unlike standard attention mechanisms that need to retain all historical keys and values, DeltaNet compresses historical information through the delta rule (an update method based on error correction) and uses gating to control write strength. This "read-modify-write" state update process is highly similar to the semantics of computer storage: the state vector can be read for computation, modified to write new information, and retained across multiple time steps. It is precisely this property that allows qwenGPU to use a single state to serve as both registers and memory—registers as fast, frequently updated short-term state, and memory as larger-capacity, longer-retained long-term state.

In a traditional Transformer, the attention mechanism has no explicit persistent state, and information transfer at each layer depends on the context window; the recursive state of Gated DeltaNet naturally has the continuity of an "execution trace," which is closer to how a hardware state machine evolves. This also explains why the article argues that state space structures may be better suited than attention mechanisms for carrying hardware state.

The model writes out every step of a CUDA launch itself

Another claim of qwenGPU is that the model itself writes out every step of a real CUDA launch. This is fundamentally different from "the model outputs a piece of CUDA code"—the former emphasizes the item-by-item generation and organization of execution steps, while the latter is only text-level code synthesis.

From an engineering perspective, this means the model's output must strictly align with the GPU's actual launch process, including kernel configuration, thread block partitioning, state initialization, and other steps. Although the original information does not disclose the specific verification method or success rate, this direction hints at a new possibility: using the parameters of a language model to carry a complete description of the hardware execution flow rather than relying on an external compiler or runtime.

Differences from the conventional code generation approach

When conventional AI coding assistants generate CUDA, they essentially transfer code patterns from training corpora to new tasks, and the model does not "understand" what happens inside the GPU. qwenGPU attempts to bypass this intermediary and encode the GPU's execution semantics directly into the weights. If this idea holds, the model's modeling of hardware behavior will no longer depend on textual descriptions but on structural parameter mappings.

However, the fact that the 45 weight layers are untrained also shows that the current stage is more of an architectural proof of concept than a directly usable inference system. Untrained means these layers have not yet undergone gradient optimization to fit real GPU behavior, and their actual execution fidelity still needs further verification.

The CUDA launch process refers to the complete steps from the host-side invocation of a kernel to the GPU's actual execution, typically including: selecting the device and stream, configuring kernel parameters, setting grid and thread block dimensions, allocating and initializing device memory, loading the kernel function onto the device, and finally triggering execution. In conventional programming, these steps are accomplished by the CUDA runtime API (such as cudaLaunchKernel) in cooperation with the compiler. qwenGPU's claim that the model writes these steps itself means the model's output needs to precisely correspond to this series of configuration and state initialization actions, rather than merely generating the source code of the kernel function body.

Compared with text code generation, item-by-item generation of execution steps requires the model to have a structured understanding of CUDA's runtime semantics, including the relationship between thread block partitioning and register pressure, the timing of shared memory allocation, and so on. This is also why the article emphasizes that the two are fundamentally different: code generation only requires syntax and pattern matching, while execution step generation requires the model to reproduce a deterministic launch state machine within its parameter space.

What this experiment means

The idea of compiling hardware into model weights touches on a fundamental question about the relationship between AI systems and underlying computing resources: can a model become an "endogenous representation" of a computing device rather than merely software running on top of it? qwenGPU's answer is still early, but it at least reveals three directions worth tracking.

First, the semantic boundaries of model weights are broadened. Weights no longer represent only language statistics but can also represent deterministic execution logic. Second, the potential of state space structures such as Gated DeltaNet for "simulation" tasks is being re-examined, and their state retention capability may be better suited than attention mechanisms for carrying hardware state. Third, the item-by-item generation of the CUDA launch process offers a new paradigm for interface design between models and GPUs.

For developers interested in the intersection of AI programming and low-level systems, qwenGPU is more of a question than a conclusion: when a model can simulate GPU execution, how will the roles of compilers and runtimes be redefined? The answer to this question may be more valuable than the 45 weight layers themselves.

One-line Summary

qwenGPU compiles a 4-lane SIMT GPU into 45 untrained weight layers of Qwen3.5-2B, using Gated DeltaNet state to simulate registers and memory.

Paragraph Summary

The qwenGPU project proposes a rather transgressive idea: instead of letting the model access the GPU through tool calls, it compiles the instruction stream and execution semantics of a 4-lane SIMT GPU directly into Qwen3.5-2B, forming 45 untrained parameter layers. Its core design uses the Gated DeltaNet state to serve simultaneously as registers and memory, so that the forward pass simulates the execution trace of a GPU kernel and the model writes out each step of a real CUDA launch itself. The article points out that this is fundamentally different from the conventional code generation approach—the latter relies on text pattern transfer, while the former attempts to reproduce a deterministic state machine in the weights. However, the untrained weights indicate that the work is still at the architectural proof-of-concept stage, and execution fidelity remains to be verified. Its value lies in broadening the semantic boundaries of model weights and re-examining the potential of state space structures for hardware simulation tasks.