Unverified60% confidenceFactExact time
Continuous batching allows dynamic adjustment of the batch at each decoding step, immediately inserting a newly waiting request once a request completes; Orca (a scheduling system proposed in 2022) was the first to implement this idea.
2
Sources
60%
Confidence
Long-term
Relevance
8/13/2026
First Seen
Sources
Related Claims
UnverifiedThe core idea of continuous batching is to dynamically replace completed requests with new ones at each iteration step, thereby improving hardware utilization.74% similarUnverifiedHybrid mode attaches a portion of samples from the next automated batch, allowing the request to arrive with recent execution context.71% similarUnverifiedTraditional static batching requires waiting for an entire batch of requests to complete before accepting new ones.71% similarUnverifiedContinuous batching checks which sequences have finished after each decoding step and dynamically inserts newly arrived requests into the freed slots, keeping the batch full at every decoding step.70% similarVerified连续批处理（Continuous Batching）采用迭代级别的调度粒度，每完成一个 token 生成便可移出已完成请求并插入新请求70% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/740519API
curl https://kongchang.com/api/v1/knowledge/claims/740519MCP
get_claim(id=740519)