[KongchangAI]
· 2 min read· 1,359 words

Apache Spark Real-Time Mode: Streaming Latency Drops to Milliseconds

Apache Spark Real-Time Mode: Streaming Latency Drops to Milliseconds

Apache Spark Structured Streaming introduces the Real-Time Mode trigger, breaking through the second-level latency bottleneck of micro-batch with millisecond-level continuous processing.

This article introduces Real-Time Mode in Apache Spark Structured Streaming. Traditional micro-batch triggers cause end-to-end latency to remain at the second level due to batch scheduling and serialization overhead, while Real-Time Mode compresses latency to milliseconds by continuously processing data and eliminating batch boundaries. It is suitable for latency-sensitive scenarios such as real-time risk control, online feature computation, real-time monitoring and alerting, and interactive applications. The article emphasizes that Real-Time Mode complements rather than replaces micro-batch: lower latency usually comes with reduced throughput and increased state management costs, and operations monitoring and resource scheduling also need adjustments. Teams should choose the appropriate trigger based on actual latency and throughput requirements.

Apache Spark's Structured Streaming has long been a key foundation for unified batch and stream processing, but in latency-sensitive scenarios, the second-level latency of the micro-batch processing model has always been a barrier. The introduction of Real-Time Mode is precisely aimed at breaking through this barrier, enabling Spark to process streaming data with millisecond-level latency.

What Is Real-Time Mode?

Real-Time Mode is a trigger for Apache Spark Structured Streaming. Traditionally, Structured Streaming uses Micro-Batch triggers, splitting incoming data into small batches at fixed intervals for processing. Each batch incurs scheduling and serialization overhead, so end-to-end latency typically remains at the second level.

Real-Time Mode changes this processing paradigm. It no longer waits for batch boundaries but continuously processes data, greatly shortening the time window from data entering the system to results being output. According to the original material, Real-Time Mode is a trigger for processing streaming data, with the core goal of achieving low-latency continuous processing.

Apache Spark Real-Time Mode illustration

The design philosophy of this trigger complements Spark's existing micro-batch engine: micro-batch is suitable for throughput-first, latency-insensitive ETL scenarios, while Real-Time Mode targets businesses requiring immediate response, such as real-time risk control, online feature computation, and real-time monitoring and alerting.

Trigger is the mechanism in Structured Streaming that controls the execution cadence of streaming queries. It determines when and how the engine converts incoming data into a processing batch. Common trigger types include: fixed-interval micro-batch (e.g., Trigger.ProcessingTime("1 second")), one-time trigger (Trigger.Once), trigger immediately when available (Trigger.AvailableNow), and the Real-Time Mode discussed in this article (typically enabled in Spark via Trigger.RealTime() or similar configuration). Triggers do not change the programming model of streaming queries—users still write logic using the DataFrame/Dataset API—but they profoundly affect job latency characteristics, resource utilization, and state update frequency. Understanding trigger selection is essentially making architectural decisions among latency, throughput, and operational cost.

Why Is Real-Time Mode Needed?

The Latency Bottleneck of Micro-Batch

Structured Streaming's micro-batch model performs excellently in throughput, but each batch must go through the complete process of task scheduling, shuffle, and state updates. Even if the batch interval is set very low, system overhead and end-to-end latency are still difficult to compress to the millisecond level. For applications that require "process upon data arrival," this latency window may mean missing the optimal decision timing.

Micro-Batch is the default and most mature execution model of Structured Streaming. It splits unbounded stream data into a series of small bounded batches, each submitted and executed as an ordinary Spark job. This design reuses Spark's mature batch processing engine and fault-tolerance mechanisms (such as lineage recomputation and checkpointing), thus providing strong consistency and high throughput. However, each micro-batch includes fixed overheads such as job submission, task scheduling, shuffle read/write, and state store access. Even if the batch interval is reduced to 100 milliseconds, end-to-end latency typically remains in the hundreds of milliseconds to seconds range. Furthermore, the lower bound of micro-batch latency is limited by the scheduler's minimum scheduling granularity and single-batch processing time, and cannot be infinitely compressed by simply reducing the interval.

The Continuous Processing Approach

The essence of Real-Time Mode is to shift Spark from "batch processing" to "continuous processing." Data does not need to wait for batch triggers but is continuously consumed and processed. This mode reduces the extra overhead caused by batch boundaries, significantly lowering latency.

Real-Time Mode processing flow

From an architectural perspective, this shift places higher demands on Spark's execution engine: finer-grained task scheduling, more efficient state management, and lower per-record processing overhead. Real-Time Mode is precisely the manifestation of these optimizations at the trigger level.

Continuous Processing is another technical approach in the stream processing domain, represented by Flink's streaming runtime and Spark's experimental Continuous Processing mode. Its core idea is: process data upon arrival, without waiting for batch boundaries; operators pass data in a pipeline fashion through long-running tasks and in-memory buffers. Compared to micro-batch, continuous processing can significantly reduce latency, but at the cost of potentially weakened fault-tolerance semantics (for example, Spark's early Continuous Processing only provided at-least-once semantics), higher implementation requirements for shuffle and state backends, and limited support for some operators. Real-Time Mode can be seen as a trigger-level optimization that moves Spark toward continuous processing while preserving the Structured Streaming API and semantic guarantees.

Applicable Scenarios for Real-Time Mode

Real-Time Mode is not intended to replace micro-batch mode but to provide a more suitable choice for specific scenarios. The following types of scenarios best demonstrate its value:

  • Real-time risk control and fraud detection: Transaction data needs to complete rule matching and risk scoring at millisecond level; latency directly affects interception effectiveness.
  • Online feature computation: When machine learning models perform online inference, features need to be updated in real time; micro-batch latency may cause features to become stale.
  • Real-time monitoring and alerting: Anomalies in system metrics or business metrics need to trigger alerts immediately upon occurrence, rather than waiting for the next batch.
  • Interactive applications: User behavior data needs immediate feedback into recommendations or search ranking; latency directly affects user experience.

In these scenarios, the latency reduction brought by Real-Time Mode is an architectural benefit, not merely the result of parameter tuning.

What to Consider When Using Real-Time Mode?

Throughput-Latency Trade-off

Lower latency often means a higher proportion of per-record processing overhead, and the system's overall throughput may be inferior to micro-batch mode. Therefore, when choosing a trigger, it is necessary to make trade-offs based on the actual business requirements for latency and throughput. For scenarios such as log aggregation and offline data lake ingestion, micro-batch remains the more economical choice.

Complexity of State Management

In continuous processing mode, state updates are more frequent, placing greater pressure on state stores and backends. When using Real-Time Mode, attention must be paid to the performance and capacity planning of the state backend to avoid state becoming a new bottleneck.

State Store is the component in Structured Streaming used to save intermediate results of stateful operations such as cross-batch aggregations, deduplication, and joins. The default implementation is HDFSBackedStateStore based on HDFS or local file system, which generates state versions and checkpoints with each batch commit. Under Real-Time Mode, the frequency of state updates increases significantly, and the read/write amplification, version management, and GC pressure of the state store all rise substantially. In production environments, it is often necessary to consider using more efficient state backends such as RocksDB, setting appropriate state TTLs, and planning the I/O capacity of the checkpoint directory; otherwise, the state layer may become a new bottleneck in the low-latency pipeline.

Ecosystem and Operations

Real-Time Mode changes the execution characteristics of jobs, so monitoring metrics, resource scheduling, and fault recovery strategies all need to be adjusted accordingly. Operations teams need to understand job behavior under the new mode to ensure stability.

Conclusion

Real-Time Mode completes the low-latency piece for Apache Spark Structured Streaming, enabling Spark to cover both high-throughput batch processing scenarios and millisecond-level real-time processing scenarios under the same API. Its emergence is not a negation of micro-batch mode but provides a finer-grained choice for stream processing architectures. For teams building real-time data pipelines, understanding the positioning and boundaries of Real-Time Mode is more important than blindly pursuing low latency.

One-line Summary

Apache Spark Structured Streaming introduces the Real-Time Mode trigger, breaking through the second-level latency bottleneck of micro-batch with millisecond-level continuous processing.

Paragraph Summary

This article introduces Real-Time Mode in Apache Spark Structured Streaming. It points out that traditional micro-batch triggers cause end-to-end latency to remain at the second level due to batch scheduling and serialization overhead, while Real-Time Mode compresses latency to milliseconds by continuously processing data and eliminating batch boundaries. This mode is suitable for latency-sensitive scenarios such as real-time risk control, online feature computation, real-time monitoring and alerting, and interactive applications. The article also emphasizes that Real-Time Mode does not replace micro-batch but complements it: lower latency usually comes with reduced throughput and increased state management costs, and operations monitoring and resource scheduling also need corresponding adjustments. Teams should choose the appropriate trigger based on actual business requirements for latency and throughput.

#Apache Spark#Structured Streaming#Real-Time Mode#stream processing#low latency#micro-batch#trigger
Share:

Related articles