Apache Spark 4.3 Released: Real-Time Mode Gains Stateful Processing

Apache Spark 4.3 has been officially released and is now available in Databricks Runtime 19. The core signal of this update is clear: Spark is pushing the real-time capabilities of stream processing to a new stage—Real-Time Mode now supports stateful processing, making scenarios that previously required additional components or complex architectures possible within Spark itself.

From Stateless to Stateful: A Key Step for Real-Time Mode

Spark's Structured Streaming has long been dominated by the micro-batch model. While latency has been continuously optimized, true real-time mode has always been constrained by the complexity of state management. Stateless operations can achieve extremely low latency, but as soon as logic requires cross-event memory—such as aggregations, deduplication, or session windows—it typically has to fall back to the micro-batch path.

Spark 4.3 brings stateful processing into Real-Time Mode, meaning the low-latency path is no longer limited to "one-shot" data transformations. For typical scenarios like risk control, real-time metrics, and event deduplication, state is essential—without state, most real-time business logic cannot be completed end-to-end. This change substantially expands the capability boundary of real-time mode and is the most noteworthy direction of evolution in version 4.3.

It is worth noting that stateful processing places higher demands on checkpoints, state storage, and failure recovery. In practice, attention must still be paid to the trade-off between state size and recovery latency.

Micro-batch model is the default execution mode of Structured Streaming: the engine periodically splits input data into small batches and triggers query plan execution batch by batch. This approach is mature in throughput and fault tolerance, but end-to-end latency is constrained by the batch interval, typically ranging from hundreds of milliseconds to seconds. Real-Time Mode, on the other hand, switches to a continuous processing path where events are processed as they arrive without batch splitting, thereby reducing latency to the millisecond level. The trade-off is that the continuous processing path has much higher requirements for state management, checkpoints, and failure recovery. Early versions only supported stateless or restricted stateful logic, which is why bringing stateful capabilities into Real-Time Mode is regarded as a key breakthrough.

Stateful processing refers to operators that need to maintain memory across events—for example, aggregations must retain accumulated results, deduplication must remember seen keys, and session windows must track window boundaries. Structured Streaming uses a State Store to persist these intermediate states, and writes state snapshots together with consumption offsets to reliable storage via checkpoints for reconstruction during failure recovery. State size directly determines checkpoint overhead and recovery latency: the larger the state, the slower the snapshot and the longer the recovery. Therefore, introducing stateful processing in Real-Time Mode poses engineering challenges beyond latency, including how to control state growth and how to persist state without blocking processing.

Databricks Runtime 19 Available in Sync

The new version is provided alongside Databricks Runtime 19, and users can directly use Spark 4.3 capabilities in that runtime. For teams already building streaming pipelines on the Databricks platform, this means a relatively smooth upgrade path without having to maintain Spark version and runtime compatibility themselves.

From the release cadence perspective, the bundled release of Spark and Databricks Runtime has become customary: new features on the Spark side are first validated for availability in the runtime and then gradually spread to broader deployment forms. For users with self-managed clusters, attention still needs to be paid to feature differences between the community edition and the distribution.

Databricks Runtime (DBR) is Databricks' packaged distribution of Apache Spark, which includes, in addition to Spark itself, an optimized execution engine, connectors, and managed service integrations. DBR has a version mapping relationship with open-source Spark, but it is not fully aligned feature by feature; some capabilities are first provided in DBR or tuned on the platform side. Therefore, "Spark 4.3 available in DBR 19" means platform users can obtain the capability earlier, while self-managed cluster users need to upgrade to the corresponding open-source version themselves and bear the compatibility verification and operational work on their own.

What This Update Means

The value of real-time stateful processing lies in architectural simplification. In the past, to implement low-latency stateful logic, a common approach was to introduce a separate stream processing engine or maintain a separate state store outside Spark. The direction of Spark 4.3 is to bring this capability back into a unified engine and reduce data movement between systems.

Of course, the release information itself is relatively brief. Specific implementation of the state backend, behavioral differences from the existing micro-batch mode, and quantitative performance data still require more detailed technical documentation and community practice feedback. For production environments, it is recommended to first verify behavioral consistency of stateful real-time mode on non-critical pipelines before gradually migrating.

Upgrade Recommendations

• Evaluate operators in existing streaming jobs that depend on state and determine whether they are suitable for migration to Real-Time Mode
• Pay attention to checkpoint compatibility and state recovery behavior, especially for jobs with large state sizes
• Test first in a Databricks Runtime 19 environment before considering production cutover
• Continuously track community practice reports on Spark 4.3 stateful real-time mode

The release of Spark 4.3 continues Spark's investment direction in real-time stream processing. Whether stateful real-time mode can stably deliver at production scale will determine the actual weight of this update.

One-line Summary

Apache Spark 4.3 brings stateful processing to Real-Time Mode, available with Databricks Runtime 19.

Paragraph Summary

Apache Spark 4.3 has been officially released and is available with Databricks Runtime 19. The core of this update is that Real-Time Mode now supports stateful processing, so the low-latency path is no longer limited to stateless data transformations and can handle logic requiring cross-event memory such as aggregations, deduplication, and session windows. This change is clearly significant for scenarios like risk control and real-time metrics, and also helps bring capabilities previously dependent on external stream processing engines back into Spark's unified engine, simplifying architecture. At the same time, stateful processing places higher demands on checkpoints, state storage, and failure recovery, and the trade-off between state scale and recovery latency remains key to production deployment. Because the original release information is relatively brief, specific implementation details and performance data still await community practice validation.