Zerobus Ingest + Debezium: A Practical Guide to Push-Based CDC into the Lakehouse

Syncing data from operational databases to the data lakehouse in real time is a critical step for many enterprises building real-time analytics capabilities. Traditional polling-based Change Data Capture (CDC) solutions often introduce latency and place pressure on source databases, whereas push-based CDC can capture data changes more efficiently. This article combines Databricks' Zerobus Ingest with the open-source tool Debezium to explore how to write CDC data into Unity Catalog in a push-based manner, and analyzes the design approach and practical value of this solution.

Why Do We Need Push-Based CDC?

Data such as orders, inventory, and user behavior in operational databases drives business decisions, but traditional analytics often relies on batch extraction, resulting in stale data. CDC technology captures database change logs (such as MySQL's binlog) to enable near-real-time data synchronization. However, common CDC pipelines use polling to pull changes, which not only increases the burden on source databases but also makes it difficult to guarantee low latency. Push-based CDC lets the source actively push change events to the target, reducing polling overhead and better aligning with the needs of modern streaming data processing.

Debezium, as a leading open-source CDC platform, can convert database changes into event streams, while Zerobus Ingest is a managed ingestion service provided by Databricks, specifically designed to efficiently push external data into Unity Catalog. The combination of the two offers a new option for building low-latency, low-maintenance CDC pipelines.

Change Data Capture (CDC) is a technique for identifying data changes by reading a database's transaction logs (rather than executing queries). Taking MySQL as an example, its binlog records all data modification operations as events. CDC tools parse these logs to obtain before-and-after data snapshots as well as the operation type (insert, update, delete). Compared with timestamp-based polling for batch extraction, CDC is less intrusive to the source database and can capture intermediate state changes (including deletes), making it the foundation for near-real-time synchronization. The core difference between push and pull lies in who initiates data flow: in push mode, the source or an intermediate layer actively sends events to the target, and the target does not need to query periodically, thereby avoiding the idle overhead and inherent latency of polling.

Core Mechanisms of Zerobus Ingest

Zerobus Ingest is the ingestion component in the Databricks Data Intelligence Platform. It supports pushing event streams from external systems directly into tables or volumes managed by Unity Catalog. Unlike traditional periodic pulling or file upload methods, Zerobus provides an API-based push interface that allows clients to continuously send data batches and automatically handles schema evolution, data versioning, and transactional writes. This means that once data arrives, it can be queried by downstream analytics engines, significantly reducing end-to-end latency.

For CDC scenarios, Zerobus's push model naturally fits Debezium's event streams. Debezium sends each change event as an independent message, and Zerobus ingests these messages in batches and commits them to Unity Catalog tables, guaranteeing exactly-once semantics. In addition, Zerobus integrates with Unity Catalog's permission system, enabling unified management of data access control.

Exactly-once semantics means that in distributed data processing, each message is processed exactly once—neither lost nor duplicated. In CDC pipelines, this guarantee is crucial because duplicate writes can inflate downstream aggregation results, while losses cause data inconsistency. Zerobus Ingest achieves this semantics by assigning a unique identifier to each push batch and combining transactional commits with idempotent write mechanisms: when a client retries, the server can identify and ignore batches that have already been successfully committed, thus avoiding duplicate data. Note that end-to-end exactly-once also depends on the configuration of upstream Debezium and Kafka (such as Kafka's idempotent producer and transaction support); Zerobus alone cannot cover the entire chain.

Debezium's CDC Capture Capabilities

Debezium deploys Kafka Connect connectors to monitor database transaction logs and convert insert, update, and delete operations into structured change events. These events contain before-and-after data as well as metadata (such as source database position and transaction ID). Debezium supports multiple databases, including MySQL, PostgreSQL, MongoDB, and more, and offers good fault tolerance and scalability.

In a push-based pipeline, Debezium is responsible for capturing changes from the source database and sending events to a message queue (such as Apache Kafka). A lightweight push client then consumes events from Kafka and pushes data to Unity Catalog via the Zerobus Ingest API. This architecture decouples CDC capture from data ingestion, preserving Debezium's flexibility while leveraging Zerobus's managed advantages.

Kafka Connect is a distributed data integration framework in the Apache Kafka ecosystem that moves data between external systems and Kafka in batch or streaming mode through "connector" plugins. Debezium runs as a Kafka Connect connector, typically deployed in a Kafka Connect cluster, where the framework handles connector lifecycle management, task parallelism allocation, and failover. After monitoring source database logs, the connector writes change events to one or more Kafka topics, with each topic corresponding to a source table. This design decouples CDC capture from downstream consumption: multiple consumers can independently subscribe to the same topic without affecting each other. During deployment, pay attention to the connector's number of worker tasks, offset storage topics, and Kafka cluster capacity planning—these factors directly affect pipeline throughput and recovery behavior.

Integration Architecture and Workflow

A typical integration workflow includes the following steps:

1. Source database change capture: Debezium connectors monitor the source database and write change events to Kafka topics.
2. Event consumption and transformation: The push client subscribes to Kafka topics and performs necessary format conversion or filtering on events.
3. Zerobus push: The client calls the Zerobus Ingest API to push batch data to the target table in Unity Catalog.
4. Table management and querying: Unity Catalog automatically manages table metadata and permissions, and downstream users can query real-time data via Databricks SQL or Spark.

The advantages of this architecture are: the push model reduces latency and resource waste caused by polling; Zerobus's managed nature lowers operational complexity; and Unity Catalog provides unified governance. At the same time, Debezium's mature ecosystem ensures source-side compatibility.

Key Considerations in Practice

Although push-based CDC has many advantages, several points require attention during implementation:

• Schema evolution and compatibility: When source table structures change, ensure that the schema evolution strategy for Debezium events and Zerobus target tables is consistent to avoid write failures.
• Error handling and retries: The push client must implement a robust retry mechanism and handle error responses returned by Zerobus to prevent data loss.
• Performance tuning: Batch size, push frequency, and concurrency need to be adjusted according to data volume to balance latency and throughput.
• Monitoring and observability: Monitor the CDC pipeline's latency, error rate, and data integrity to ensure data quality.

Use Cases and Limitations

The push-based CDC-to-lakehouse solution is suitable for scenarios with high requirements for data freshness, such as real-time risk control, dynamic pricing, and operational dashboards. For systems with small data volumes or low change frequency, traditional batch synchronization may be more cost-effective. In addition, this solution requires source-side CDC support (such as enabling binlog on the database) and the maintenance of intermediate components like Kafka, which imposes certain requirements on the team's technology stack.

From a trend perspective, with the proliferation of data lakehouse architectures, push-based ingestion is becoming an important bridge connecting operational systems and analytics platforms. The combination of Zerobus and Debezium provides an actionable reference implementation for this model. In the future, we can expect more managed services to simplify the construction of CDC pipelines, further lowering the barrier to real-time data synchronization.

One-line Summary

Combine Zerobus Ingest with Debezium to achieve low-latency, push-based CDC data writes into Unity Catalog.

Paragraph Summary

The article explores how to synchronize change data from operational databases to Databricks Unity Catalog in real time using a push-based approach. Traditional polling-based CDC suffers from high latency and heavy source database load, whereas push-based CDC has the source actively send change events, better aligning with streaming processing needs. Debezium captures changes from the source database and writes them to Kafka, and a lightweight client then pushes batch data to Unity Catalog tables via the Zerobus Ingest API. Zerobus provides a managed push interface that automatically handles schema evolution, versioning, and transactional writes while guaranteeing exactly-once semantics. Architectural advantages include reduced latency, lower operational burden, and unified governance, but implementation requires attention to schema compatibility, error retries, performance tuning, and monitoring. This solution is suitable for scenarios with high data freshness requirements, such as real-time risk control and dynamic pricing.