Micro-Batch
Micro-Batch is the default execution model of Apache Spark Structured Streaming. It divides unbounded stream data into small bounded batches for processing, reusing the Spark batch processing engine and fault-tolerance mechanisms. It provides strong consistency and high throughput but incurs second-level latency overhead.
Timeline (last 90 days)
Structured Streaming's micro-batch trigger splits incoming data into small batches for processing at fixed intervals, and end-to-end latency is typically on the order of seconds.
Micro-batch mode suits ETL scenarios that prioritize throughput and are insensitive to latency, while Real-Time Mode targets businesses that need immediate response (such as real-time risk control, online feature computation, and real-time monitoring and alerting).
In the micro-batch model, each batch incurs fixed overheads such as job submission, task scheduling, shuffle reads and writes, and state store access; even if the batch interval is tuned to 100 milliseconds, end-to-end latency is still typically in the range of hundreds of milliseconds to seconds.
The latency lower bound of micro-batch is limited by the scheduler's minimum scheduling granularity and the processing time of a single batch, and cannot be compressed indefinitely by merely reducing the interval.
For scenarios such as log aggregation and offline data lake ingestion, micro-batch remains the more economical choice.
All Facts (5)
Structured Streaming's micro-batch trigger splits incoming data into small batches for processing at fixed intervals, and end-to-end latency is typically on the order of seconds.
50%UnverifiedMicro-batch mode suits ETL scenarios that prioritize throughput and are insensitive to latency, while Real-Time Mode targets businesses that need immediate response (such as real-time risk control, online feature computation, and real-time monitoring and alerting).
50%UnverifiedIn the micro-batch model, each batch incurs fixed overheads such as job submission, task scheduling, shuffle reads and writes, and state store access; even if the batch interval is tuned to 100 milliseconds, end-to-end latency is still typically in the range of hundreds of milliseconds to seconds.
50%UnverifiedThe latency lower bound of micro-batch is limited by the scheduler's minimum scheduling granularity and the processing time of a single batch, and cannot be compressed indefinitely by merely reducing the interval.
50%UnverifiedFor scenarios such as log aggregation and offline data lake ingestion, micro-batch remains the more economical choice.
50%