Batching

Increasing batch size to improve throughput typically sacrifices per-request latency

The randomness of LLM inference comes from at least three independent sources: the sampling strategy (temperature, top-p), the accumulated error of floating-point operations, and changes in the reduction order of matrix multiplications caused by changes in batch size.

Batch processing suits latency-insensitive, high-throughput scenarios, and is simple to implement and lower in cost.

The application demands of real-time analytics are compressing the decision window from hours to seconds or even sub-seconds.

A practical way to judge whether to introduce a real-time architecture is to look at the decision window: if minutes or more are acceptable, optimizing the batch scheduling frequency may be more cost-effective; only when the decision window is compressed to seconds and the benefits cover the extra complexity and cost is real-time analytics worth investing in.

The cost curves of continuously running computation tasks and on-demand scheduled batch tasks are completely different, and the cost of real-time systems often only becomes apparent after they run stably.

The core assumption of traditional batch processing is that decisions can wait, and when facing events that require immediate response, latency becomes a hard flaw.

Content freshness and permission freshness do not need the same solution: one needs a faster batch job, and the other needs to stop doing batch jobs entirely.

Batch processing handles accumulated data in fixed time windows, while stream processing responds immediately as data arrives; the two differ significantly in tool selection, cost structure, and operational complexity.

For inference service providers, aggressive batching strategies can improve throughput but may increase latency for individual requests.

In Coze workflows, loop nodes are equivalent to a single machine processing items one by one, while batch processing is equivalent to multiple machines processing simultaneously—batch processing is more efficient but consumes more resources and has more restrictions.

Inputs of different lengths within a batch need to be padded to a uniform length, which introduces additional computational overhead.

A complete batch task management lifecycle typically includes five phases: creation, querying, monitoring, cancellation, and result retrieval.

The lack of batch cancellation capability means that once a task containing an erroneous prompt has been submitted, one can only wait for it to complete, wasting computational resources and API quota.

Batch processing bundles large numbers of requests for submission, processes them asynchronously on the server side, and returns results collectively, usually accompanied by more favorable pricing.

Modern large model inference typically uses batching technology to improve GPU utilization, and backlogs in the batching queue during peak periods cause the time to first token (TTFT) to become longer.

When an inference service serves a large number of concurrent requests simultaneously, batching is the core means of driving down per-unit costs.

When batching 8 requests together, a single RTX PRO 6000 can achieve an aggregate throughput of approximately 220 output tok/s, whereas the old configuration of two RTX 3090s achieved only approximately 19 tok/s.