No Kafka Required: A Practical Guide to Building Real-Time IoT Architecture with Databricks

What Do You Do When Thousands of Devices Are Pushing Events Simultaneously?

Massive concurrent device data reporting is one of the most fundamental challenges in IoT systems. Picture this: thousands of phones and sensors simultaneously pushing event streams to a backend. Traditional architectures typically require complex message queue systems to absorb that flood of data.

A recent demo project offers a cleaner approach: bypassing Kafka and Kinesis entirely, and building an end-to-end real-time IoT architecture using only components within the Databricks ecosystem. According to the demo, it can handle real-time sensor data reported by phones worldwide, process thousands of concurrent IoT events, and do so without any Kafka or Kinesis deployment or operations.

For teams long burdened by the configuration, scaling, and maintenance of message brokers, this is a simplification worth taking seriously.

Three Core Components of the Architecture

This real-time IoT architecture consists of three parts, each playing a distinct role, together forming a complete closed loop from data ingestion to front-end display.

Zerobus Ingest: High-Concurrency Data Ingestion Layer

The entry point of the architecture is Zerobus Ingest, designed specifically to handle event streams from massive numbers of devices. It plays the role that Kafka/Kinesis fills in traditional solutions — acting as a high-throughput ingestion layer that consolidates sensor data from around the world.

As shown above, the demo provides real-time visualization of data ingested by ZeroBus, offering an intuitive view of the data flow state. Eliminating the need to build your own message queue means developers no longer have to worry about complex concepts like partitions, consumer groups, or offset commits — the ingestion layer is dramatically simplified.

Lakebase Postgres: Low-Latency Data Serving Layer

Once data arrives, how to serve it quickly for queries is another critical piece. This architecture introduces Lakebase Postgres as the low-latency serving layer.

As the demo puts it, "we've added an extra lakebase here" — this layer is dedicated to serving low-latency queries. Postgres is a widely proven relational database, and combined with Lakebase's positioning, it preserves a familiar SQL interface while meeting low-latency performance requirements. It serves as the "fast lane" for application queries across the entire pipeline.

Databricks Apps: Unified Application Layer for Front-End and Back-End

At the top of the architecture sits Databricks Apps, handling both front-end display and back-end logic. From data visualization interfaces to business logic processing, everything can be accomplished within the Databricks platform — no need to stand up a separate web service or introduce another application framework.

The value of this unified design lies in convergence: device reporting, ingestion, storage serving, and final display all live within the same platform ecosystem, significantly reducing the friction of cross-system integration.

Where Exactly Is This Architecture "Simpler"?

The keyword that comes up repeatedly in the demo is simpler. Compared to traditional real-time data architectures, the simplification shows up in several key areas:

• No message queue operations: No need to deploy and maintain a Kafka cluster or configure Kinesis streams — which is often the heaviest operational burden in real-time architectures.
• Component consolidation: Ingestion, serving, and application layers all live within the Databricks ecosystem, so the technology stack is no longer fragmented.
• Familiar technology: Using Postgres as the serving layer means teams can directly reuse existing SQL skills and tooling.

For small-to-medium teams or developers looking to quickly validate IoT prototypes, this lower barrier to entry is especially appealing. Processing thousands of concurrent events and ingesting real-time sensor data from phones worldwide doesn't require building complex stream-processing infrastructure from scratch.

Performance: A Trade-Off Worth Acknowledging

The demo is also honest about the real-world experience. During one operational segment, the presenter flatly stated, "this isn't actually that fast."

This is a reminder that every "simplified" architectural choice comes with trade-offs. By bypassing specialized stream-processing middleware, you gain deployment simplicity, but in certain extreme performance scenarios, it may fall short of a highly tuned Kafka/Kinesis setup. Before moving to production, teams still need to conduct targeted evaluations and load testing based on their specific latency requirements and concurrency scale.

Note: The performance observations above are drawn from the live experience of a single demo video, which represents a limited sample. Always rely on official benchmarks and your own measured data.

Conclusion: A New Path for Real-Time IoT Architecture

This Databricks-based real-time IoT architecture offers a "less is more" solution. The combination of Zerobus Ingest, Lakebase Postgres, and Databricks Apps compresses what would normally require multiple independent systems working together into a single unified platform.

Its significance isn't about replacing Kafka/Kinesis in every scenario — it's about providing teams that don't want to bear the operational cost of heavy-duty stream processing with a more accessible on-ramp. For rapid prototype validation and moderate-scale real-time IoT applications, this is a solution worth adding to your shortlist. Whether to ultimately adopt it still comes down to real-world trade-offs around specific latency, throughput, and cost requirements.