Batch Inference

Batch Transform does not run as a persistent service; it completes inference on a batch of data at once and then releases resources, suitable for scenarios such as offline scoring and periodic reporting, and is usually lower in cost.

Batch inference refers to passing multiple independent tasks in a single model call at the same time, where the model must return results corresponding one-to-one with the inputs, which differs from the concept of batch size in traditional machine learning.

Batch inference requires the output structure to be strictly aligned with the input order, and the model must not reduce its discriminative power for individual items as the context grows longer.

Batch mode compresses the number of requests to a single digit, which can bypass the RPM bottleneck and amortize the fixed overhead of the system prompt, reducing the average cost per item.

Online inference requires low latency and high concurrency, and vendors must reserve GPU resources to handle burst traffic, resulting in a higher unit cost; batch inference allows tasks to be queued and executed during off-peak periods, resulting in a lower unit price.

Price cuts in the batch inference tier usually appear after vendor scheduling algorithm optimizations, inference framework upgrades (such as vLLM, TensorRT-LLM, etc.) or the deployment of a new generation of GPUs.

Batch inference combines multiple user requests into a single batch to be processed in one GPU forward pass, thereby amortizing fixed overhead and improving hardware utilization.

When the same set of weights serves requests with different numbers of recurrent passes, the varying completion times of different requests become misaligned, leveling out the apparent throughput speed across different model tiers.