Flide Engine: Exact State Forking Delivers 9.5x Speedup for Decision Reasoning

Decision models (such as Jev) are emerging as a highly promising paradigm in LLM agent systems, providing lightweight and reliable classification reasoning for fast "System 1" execution. However, existing generation-oriented inference engines cannot effectively support this type of workload — they either face severe prefill recomputation overhead or suffer from distorted output probabilities. To address this challenge, researchers propose Flide — an efficient and exact inference engine that decouples the logical tree structure of decision requests from physical execution on a single flattened token stream.

The Dilemma of Decision Inference: Recomputation and Probability Distortion

Traditional LLM inference engines (such as vLLM and SGLang) are optimized for autoregressive generation, with a core assumption that requests are independent of each other and sequences expand unidirectionally. But decision models work differently: they need to evaluate multiple branches within the same context (for example, an agent faces multiple candidate actions at each step), forming a tree-structured reasoning pattern. Processing each branch individually with existing engines leads to massive redundant computation of shared prefixes; attempting to share prefixes may distort output probabilities due to improper attention masking or state management, undermining decision reliability.

Flide's starting point is precisely to resolve this contradiction: preserving the mathematical exactness of branch evaluation while eliminating redundant computation.

Prefill Recomputation: In autoregressive LLM inference, the prefill phase processes the input prompt and builds the KV cache. When multiple decision branches share the same prefix, if each branch independently performs prefill, the attention computation and KV cache construction for the shared prefix are repeatedly executed, causing significant waste of compute and increased latency. The more branches and the longer the shared prefix, the greater the redundant overhead.

Attention Masking and Probability Distortion: In tree-structured reasoning, tokens from different branches should not attend to each other. If branches are simply concatenated into a long sequence with shared prefix caching, attention masks must prevent cross-branch information leakage. Improper mask design or state management errors can cause certain queries to unexpectedly attend to tokens from other branches, thereby altering the softmax normalization results and causing output probabilities to deviate from true values. For decision models that rely on probabilities for classification or scoring, this distortion directly undermines decision reliability.

ForkAttention: Exact Forking in GPU Indexes

Flide's core innovation is ForkAttention. Rather than treating each branch as an independent request, it encodes the entire logical tree structure of a decision request as GPU-resident indexes and executes on a single flattened token stream. For both standard attention layers and recurrent Gated DeltaNet layers, ForkAttention can navigate request dependencies based on these indexes, ensuring that each query branch's computation results are exactly identical to an isolated forward pass — in the paper's words, "guaranteeing that each query branch is evaluated identically to an isolated forward pass under exact arithmetic."

The key to this design is that branch topology is directly expressed as GPU metadata, rather than by duplicating computation graphs or reorganizing batches. As a result, the engine can accommodate diverse request mixes with a compact set of pre-recorded execution graphs, while allowing growing agent histories to resume from checkpoints saved in earlier passes, avoiding recomputation from scratch.

Gated DeltaNet Layer: Gated DeltaNet is a linear attention variant belonging to the family of recurrent state models. It uses gating mechanisms to control state updates, maintaining a compact hidden state in a recurrent manner, thereby achieving linear complexity over long sequences. Unlike standard attention, it does not retain a full KV cache but instead updates state recursively at each time step. In tree-structured reasoning, each branch requires independent recurrent state evolution, so ForkAttention must precisely navigate branch dependencies to ensure that each branch's state update order is consistent with an isolated forward pass. This is also the key to the work covering hybrid models (attention + recurrent layers).

Pre-recorded Execution Graphs: CUDA Graph records a series of GPU kernel launches as a static graph, submitted at once at runtime, eliminating the CPU overhead of per-kernel launches. Flide leverages compact pre-recorded execution graphs to accommodate diverse request mixes, meaning requests with different tree structures can be mapped onto a small number of precompiled computation graphs for execution, thereby avoiding the overhead of dynamically building computation graphs for each request while maintaining low latency.

Performance: Up to 9.5x Speedup and Closer to the Floating-Point Reference

Extensive evaluations on publicly available hybrid models and decision models show that Flide answers queries up to 9.5x faster than vLLM and SGLang, while being closest to the float32 reference in total variation distance. This means it not only leads significantly in speed but also outperforms comparison engines in output probability exactness — which is crucial for agent applications that rely on probabilities for decision-making.

Interestingly, this speedup does not come at the expense of exactness. Flide unifies physical execution throughput with mathematical exactness, establishing a principled foundation for scalable, latency-sensitive decision intelligence in agent systems.

Total Variation Distance: A metric measuring the difference between two probability distributions, defined as half the sum of the absolute differences in probabilities across all events, ranging from [0,1], where 0 indicates perfect agreement. When evaluating the output exactness of inference engines, the probability distribution obtained from a high-precision float32 forward pass is used as the reference, and the total variation distance between the engine's output and the reference is computed. A smaller distance indicates higher fidelity of the engine's probabilities when executing under limited precision (such as fp16/bf16). Flide is closest to the float32 reference on this metric, indicating that it introduces no additional probability bias while accelerating — which is crucial for the reliability of decision models.

Implications for Agent Systems

The rise of decision models (such as Jev) reflects a differentiation in agent architectures: System 1-style fast classification reasoning and System 2-style deliberate generation each serve their own roles. Flide provides a dedicated engine for the former, filling a gap at the infrastructure level. As agent histories continue to grow and branching decisions become increasingly frequent, inference engines capable of exactly and efficiently forking states and reusing checkpoints will become key components for building low-latency agent applications.

From a broader perspective, Flide's approach — decoupling logical structure from physical execution and using metadata to drive GPU computation — may also inspire other LLM workloads that require tree-structured or graph-structured reasoning, such as Tree of Thoughts or parallel tool calling. However, the paper currently focuses on decision models, and the generality of this approach remains to be further validated.

One-line Summary

Flide inference engine precisely forks tree-structured decision reasoning via ForkAttention, achieving up to 9.5x speedup with more exact probabilities.

Paragraph Summary

Flide is an efficient and exact inference engine for LLM decision models (such as Jev). Traditional engines are optimized for autoregressive generation and either recompute shared prefixes or distort output probabilities due to improper attention masking when handling tree-structured decision requests. Flide's core innovation, ForkAttention, encodes the logical tree structure of decision requests as GPU-resident indexes and executes on a single flattened token stream, making each branch's computation results exactly identical to an isolated forward pass while supporting both standard attention and Gated DeltaNet layers. On public hybrid and decision models, Flide is up to 9.5x faster than vLLM and SGLang, with total variation distance closest to the float32 reference. This work provides infrastructure that balances throughput and exactness for latency-sensitive agent decision inference.