Flide
Flide is an efficient exact inference engine for decision models. Its core innovation, ForkAttention, encodes the logical tree structure as GPU indices and exactly forks states over a flattened token stream. It supports standard attention and recurrent Gated DeltaNet layers, eliminates prefill recomputation and remains probabilistically exact.
Timeline (last 90 days)
Flide is an efficient and accurate inference engine that decouples the logical tree structure of decision requests from physical execution in a single flattened token stream.
The core innovation of Flide is ForkAttention, which does not treat each branch as an independent request but instead encodes the logical tree structure of the entire decision request as a GPU-resident index and executes on a single flattened token stream.
Flide uses compact pre-recorded execution graphs to accommodate diverse request mixes, meaning that requests with different tree structures can be mapped onto a small number of precompiled computation graphs for execution, thereby avoiding the overhead of dynamically building a computation graph for each request while maintaining low latency.
Extensive evaluations on public hybrid models and decision models show that Flide answers queries up to 9.5 times faster than vLLM and SGLang.
In terms of total variation distance, Flide is closest to the float32 reference, outperforming the comparison engines.
The paper currently focuses on decision models, and the generality of the Flide method remains to be further verified.
All Facts (6)
Flide is an efficient and accurate inference engine that decouples the logical tree structure of decision requests from physical execution in a single flattened token stream.
50%UnverifiedThe core innovation of Flide is ForkAttention, which does not treat each branch as an independent request but instead encodes the logical tree structure of the entire decision request as a GPU-resident index and executes on a single flattened token stream.
50%UnverifiedFlide uses compact pre-recorded execution graphs to accommodate diverse request mixes, meaning that requests with different tree structures can be mapped onto a small number of precompiled computation graphs for execution, thereby avoiding the overhead of dynamically building a computation graph for each request while maintaining low latency.
50%UnverifiedExtensive evaluations on public hybrid models and decision models show that Flide answers queries up to 9.5 times faster than vLLM and SGLang.
50%UnverifiedIn terms of total variation distance, Flide is closest to the float32 reference, outperforming the comparison engines.
50%UnverifiedThe paper currently focuses on decision models, and the generality of the Flide method remains to be further verified.
50%