On-Demand State Repartitioning
A runtime feature introduced by Databricks for Apache Spark Structured Streaming that allows users to dynamically adjust the number of state partitions in stateful streaming queries without rebuilding checkpoints.
Timeline (last 90 days)
Databricks has launched the On-Demand State Repartitioning feature, which allows dynamic adjustment of the state partition count for stateful streaming queries without rebuilding the Checkpoint or losing existing state.
The implementation of on-demand state repartitioning typically introduces a repartitioning micro-batch, within which a shuffle re-routing of all keys in the State Store is performed while guaranteeing atomicity.
Triggering repartitioning causes a brief spike in latency for that batch, and operations teams need to reserve a corresponding processing window.
On-demand state repartitioning allows teams to go live first with a reasonable initial value, then gradually optimize the partition configuration based on real operational data, shifting capacity planning from a one-time decision to continuous adjustment.
All Facts (4)
The implementation of on-demand state repartitioning typically introduces a repartitioning micro-batch, within which a shuffle re-routing of all keys in the State Store is performed while guaranteeing atomicity.
50%UnverifiedTriggering repartitioning causes a brief spike in latency for that batch, and operations teams need to reserve a corresponding processing window.
50%UnverifiedOn-demand state repartitioning allows teams to go live first with a reasonable initial value, then gradually optimize the partition configuration based on real operational data, shifting capacity planning from a one-time decision to continuous adjustment.
50%UnverifiedDatabricks has launched the On-Demand State Repartitioning feature, which allows dynamic adjustment of the state partition count for stateful streaming queries without rebuilding the Checkpoint or losing existing state.
50%