Databricks at VLDB 2026: A Deep Dive into Agent-Native Data Infrastructure

Databricks' Agent-Native Vision

VLDB 2026 is underway, and Databricks — attending as a gold sponsor — has unveiled a series of forward-looking technical innovations. The data lakehouse company is redefining what data infrastructure looks like, centered on a core concept: building "Agent-Native" data systems.

Databricks co-founder Reynold Xin laid out in his keynote why the Agent era demands Lakebase Postgres. Behind this assertion lies a profound insight into a fundamental shift in AI application architecture — when LLM-driven agents become the primary participants in data processing, traditional database architectures face not an optimization challenge, but a paradigm reconstruction.

The concept of "Agent-Native" follows the natural evolution of software architecture design. Traditional data systems are built around human users — SQL is a human-readable declarative language, query plan caching is based on patterns of human repetition, and permission models assume users exercise active, intentional judgment. In an LLM-centric agent architecture, however, the callers of a database are autonomous AI programs: they dynamically generate queries at extremely high frequencies, repeatedly read and write state across multi-step reasoning chains, execute multiple subtasks concurrently, and cannot plan ahead or manually intervene in anomalies the way a human DBA can. This demands that data systems be redesigned at the level of interface semantics, concurrency control, error recovery, and observability — not simply layered with AI features on top of existing architectures.

Three Core Technical Breakthroughs: Query Optimization, Real-Time Processing, and Unified Architecture

At this year's VLDB conference, Databricks spotlighted three major technical directions, each pointing directly at the data processing demands of the Agent era.

A Revolution in Query Optimization

AutoLiquid and Ultron represent the next generation of query optimizer technology. AutoLiquid delivers automated liquid query optimization, dynamically adjusting execution strategies based on workload characteristics rather than relying on static rules. Ultron goes further, deeply integrating machine learning into the query plan generation process, enabling the optimizer to continuously learn from historical execution data and evolve autonomously.

Their shared goal is clear: when queries originate from AI agents rather than human DBAs, the optimizer must possess far greater adaptive capability.

The history of query optimizer evolution can be divided into three phases: Rule-Based Optimization (RBO) relies on expert-defined heuristic rules; Cost-Based Optimization (CBO) introduces statistical information and cost models to enumerate execution plans; and learned optimizers — exemplified by Ultron — attempt to replace or augment the cost estimation module with machine learning models. The academic community has explored this direction extensively, including deep reinforcement learning approaches like DQ (Deep Q-Network) for join order selection, and systems like Bao that use graph neural networks to model query structure. The engineering challenge lies in the limited generalization ability of ML models beyond their training distribution, while the diversity of production queries far exceeds offline training data. AutoLiquid attempts to address this through a continuous online learning mechanism.

The Three Pillars of Lakehouse Architecture

Lakehouse//RT (real-time lakehouse), Lakebase, and LTAP (Lakehouse Transactional-Analytical Processing) form Databricks' complete technology stack:

• Lakehouse//RT focuses on low-latency streaming data processing to meet the demands of real-time decision-making scenarios
• Lakebase provides a PostgreSQL-compatible storage engine, reducing migration and learning costs
• LTAP breaks down the barrier between transactional processing and analytical queries, enabling a truly unified data platform

Together, these three pillars answer one fundamental question: what kind of data foundation does an AI agent need? The answer is a converged platform that simultaneously supports transactions, analytics, and real-time stream processing.

LTAP (Lakehouse Transactional-Analytical Processing) is a lakehouse-native extension of the HTAP (Hybrid Transactional/Analytical Processing) concept. Traditional HTAP systems maintain both row-store and column-store data replicas within the same database engine to enable parallel transactional and analytical workloads — a path taken by Oracle, TiDB, and SingleStore. LTAP combines this idea with open table formats (such as Delta Lake): transactional writes land directly on a shared data layer in object storage, while analytical queries scan the same data through a vectorized engine — no ETL required. This design eliminates data redundancy and reduces consistency maintenance costs, but places greater demands on the storage layer's concurrency control and metadata management. This engineering context is also part of the reason Lakebase chose to deeply bind to the PostgreSQL-compatible protocol.

Deep Dialogue with the Academic Community

At VLDB, Databricks isn't just showcasing products — it places significant value on engaging with the database research community. The on-site dialogue sessions focused on a key proposition: when queries are no longer written by humans but automatically generated by AI, what kinds of interfaces and optimization strategies do databases need?

Upcoming technical sessions cover multiple areas:

• An in-depth architectural breakdown of Lakebase
• The latest advances in Apache Spark Structured Streaming
• Practical applications of the Enzyme compilation optimization technique

These topics reflect Databricks' continued deep investment at the systems level, and provide the research community with rich engineering practice references.

Apache Spark Structured Streaming is the unified stream-batch processing framework introduced in Spark in 2016. Its core design philosophy abstracts stream data processing as continuous queries over unbounded tables, with two underlying execution modes: Micro-Batch and Continuous Processing. Compared to the earlier Spark Streaming, Structured Streaming provides end-to-end exactly-once semantics and unifies with the DataFrame/Dataset API, dramatically reducing the complexity of stream-batch unified development. In the Agent era, the importance of stream processing is further amplified — real-time perception for AI agents, state maintenance across multi-turn conversations, and low-latency computation of online features all depend on high-throughput, low-latency stream processing. This is the fundamental reason why Lakehouse//RT occupies a standalone pillar in Databricks' overall architecture.

Reading the Trends: Where Is Data Infrastructure Headed in the Agent Era?

Databricks' technical roadmap reveals three important trends in data infrastructure:

Trend 1: Agent-First Design

Traditional databases assume users are proficient in SQL, but in the LLM-driven agent era, databases need to understand natural language intent, automatically generate optimized queries, and provide interpretable execution plans. This is not a feature-level enhancement — it is a shift in design philosophy.

Trend 2: Continued Deepening of Lakehouse Convergence

The launch of Lakebase signals that simple storage-compute separation is no longer sufficient. The market demands a unified platform that can simultaneously guarantee transactional consistency, support large-scale analytics, and handle real-time streaming data. The lakehouse paradigm is moving from concept to mature production deployment.

Trend 3: Optimizers Shifting from Rule-Driven to Learning-Driven

The adaptive optimization technologies represented by AutoLiquid and Ultron herald a paradigm migration in query optimization — from rule-based to learning-based. The optimizer itself is becoming a continuously evolving intelligent system.

Implications for Data Engineers

For data engineers and architects, the innovations Databricks showcased at VLDB 2026 mean that platform selection criteria need to be updated. Beyond traditional performance metrics and cost considerations, a system's native support for AI workloads is becoming a critical evaluation dimension.

Specifically, key factors to assess include: whether the database supports agent-friendly interface design, the adaptive learning capabilities of the query optimizer, and the degree of convergence the platform achieves across transactional, analytical, and real-time processing workloads.