[KongchangAI]
Concept微批 / Micro-Batch

Micro-Batch

Micro-Batch is the default execution model of Apache Spark Structured Streaming. It divides unbounded stream data into small bounded batches for processing, reusing the Spark batch processing engine and fault-tolerance mechanisms. It provides strong consistency and high throughput but incurs second-level latency overhead.

Timeline (last 90 days)

Nov 16

Structured Streaming's micro-batch trigger splits incoming data into small batches for processing at fixed intervals, and end-to-end latency is typically on the order of seconds.

Unverified50%
Nov 16

Micro-batch mode suits ETL scenarios that prioritize throughput and are insensitive to latency, while Real-Time Mode targets businesses that need immediate response (such as real-time risk control, online feature computation, and real-time monitoring and alerting).

Unverified50%
Nov 16

In the micro-batch model, each batch incurs fixed overheads such as job submission, task scheduling, shuffle reads and writes, and state store access; even if the batch interval is tuned to 100 milliseconds, end-to-end latency is still typically in the range of hundreds of milliseconds to seconds.

Unverified50%
Nov 16

The latency lower bound of micro-batch is limited by the scheduler's minimum scheduling granularity and the processing time of a single batch, and cannot be compressed indefinitely by merely reducing the interval.

Unverified50%
Nov 16

For scenarios such as log aggregation and offline data lake ingestion, micro-batch remains the more economical choice.

Unverified50%

All Facts (5)

Source Articles