Decoding Databricks' AI Strategy: The Omnigents Orchestration Platform and LTAP Unified Storage Architecture

In the latest episode of the Latent Space podcast, Databricks co-founder and CTO Matei Zaharia, along with core lead Reynold Xin, offered an in-depth look at the company's two differentiated bets for the AI era: the open-source Agent orchestration platform Omnigents, and the unified storage architecture LTAP.

Their core conviction is simple but powerful: "Once you can get your data into the right place, the AI models are already pretty good. A lot of traditional software will be rewritten with this new paradigm—get the data in place, connect an Agent, and the magic happens."

This article, based on the podcast, distills Databricks' technical vision and engineering practices for the future of AI.

Omnigents: A "Universal Protocol Layer" for the Agent Era

As coding Agents exploded in popularity, Databricks was among the first to hit a set of common pain points internally. Matei noted that the company runs multiple Agent products simultaneously: Isaac, a development tool built on top of Claude Code and Codex; Genie, a data science Agent created by the research team; and a large number of custom Agents built for customers. All of these systems ran into the same wall—every few months they needed to switch models and runtime frameworks (harnesses), and an Agent lacking session sharing, history preservation, search, and collaboration capabilities is essentially crippled.

Reynold's personal experience is vivid: for a whole week he wrote code from morning to night, even propping his laptop on the passenger seat while driving to a doctor's appointment, connecting via his phone's hotspot and checking the Agent's progress at every red light. "It felt like going back to the dark ages of programming." It was precisely this absurd experience that pushed the team to build a persistently running cloud sandbox environment that spins up in seconds.

At its core, Omnigents establishes a unified universal API on top of all harnesses. As Matei explains: "The API is simple—you have an Agent session, you can send messages or files, and then receive streaming output as text or tool calls." The key value is that Databricks maps Claude Code, Codex, the OpenAI SDK, and more all to the same interface. So when a model's API changes, you don't have to maintain your own adaptation layer, and you finally escape the nightmare of lost messages.

Matei likens this architecture to network protocols: "It reminds me of internet protocols and the communication standards between participants." In an era when anyone can vibe-code at will, why do we still need carefully designed protocols? The answer: when multiple participants evolving at different speeds need to coordinate, you still need a stable interoperability layer—Agents talking to Agents, users talking to Agents, tool calls—none of it works without such a specification.

Security Design: From Binary Switches to Stateful Policies

Matei is especially excited about Omnigents' security architecture. Traditional coding Agents' security controls are too coarse—you can only "allow" or "deny" a certain tool pattern. But real-world scenarios are full of dilemmas: Should the Agent read a confidential document? Should it install a freshly published NPM package that's only a day old and possibly compromised? Should it push content directly to the company's official website?

Databricks' solution is "contextual policies"—a stateful policy engine that continuously tracks the behavioral chain of an entire session. "The point isn't whether it can push to the marketing website—it's that if it did something dangerous beforehand, like installing an NPM package published yesterday, or reading thousands of confidential documents, then it's not allowed; otherwise it might be fine." This dynamic trade-off makes the system both safer and more usable at the same time.

Because policies are stateful, token budgets also become a trackable dimension. Matei gives an example: he once had an Agent debug an issue, and it ended up reading massive log files, burning through $500 outright. Now he can simply command: "Spin up a sub-Agent to handle this, cap it at $5, and come ask me if you need more." This kind of cost control is especially critical for large enterprises with tens of thousands of engineers—an extra $1,000 per person per month becomes catastrophic when multiplied by scale.

Omnigents has now been officially open-sourced. Within days of its release it received around 400 merge requests, about half from the external community, covering Kubernetes support, multi-cloud sandbox adaptation, and integration of additional Agent harnesses such as the Cursor CLI.

LTAP: Getting the HTAP Holy Grail "Right"

The second major bet is LTAP, led by Reynold. To understand it, you first need to understand a structural divide in the database world that has lasted for decades.

Databases fall roughly into two camps: OLTP (transactional, such as Postgres, MySQL, Oracle) for row-level real-time operations, and OLAP (analytical) for large-scale batch reasoning. Nearly every application starts with OLTP, but once data volumes grow and analytical needs become complex, data must be copied from OLTP to an analytics system—a process that relies on CDC (Change Data Capture).

Reynold pulls no punches on CDC: "It should be called continuous data corruption." Once a schema is modified on the OLTP side, the CDC pipeline can break, forcing countless data engineers to wake up at 3 a.m. to firefight. He even notes that Airbyte built a $5 billion company on CDC—proof of how foundational and how painful it is.

About a decade ago, the industry proposed HTAP (Hybrid Transactional/Analytical Processing)—using a single database to handle both workloads at once. But in practice, it proved to be "the worst of both worlds": the new systems couldn't inherit the vast ecosystems of Postgres or Spark, and in terms of performance they struggled to compete with specialized systems.

LTAP is a reinterpretation of HTAP—and the version Databricks believes gets it "right." The core insight: don't force unification at the query layer; unify only the storage layer. "You can get 99% of the value of the HTAP holy grail just by unifying storage, using a single storage layer." Once a Postgres database writes data in columnar format, all analytics engines can read it directly—no pipelines, zero latency.

A Strategic Breakthrough Without a "Kickoff Meeting"

The way this strategic feature came into being is surprising. Reynold revealed that he and CEO Ali spent a great deal of time arguing in meetings over "whether this is even possible," but never held a formal kickoff meeting or wrote a design document. In the end, "a particularly smart engineer just said: let me build a prototype—and it worked."

On the technical side, the team discovered that storage clusters had plenty of idle CPU that could be used to transcode row-based data (favorable for OLTP) into columnar data (favorable for analytics) in real time. After transcoding, the data compresses better, so writing to object storage like S3 is actually faster—with almost no additional overhead.

Matei summed up this culture of innovation: "If I had just written a document saying 'we could do all these things together,' everyone would start pressing me on all kinds of edge cases. But if you actually go try it, everything suddenly becomes clear." He emphasized that despite Databricks' large scale, it maintains a core culture of "you don't need to ask permission, don't go through lengthy processes, just build it first"—keeping the organization agile by hiring top talent, having founders roll up their sleeves, and strictly limiting the number of products.

Dream Engine: Rewriting the Database Engine with Machine Learning

Behind LTAP lies an even grander vision—the Dream Engine (internally nicknamed "Reynold's Dream Engine" for a while so it wouldn't easily get axed).

Reynold candidly admits that almost every analytical database engine is more than a decade old, originally designed for a specific narrow use case. As their ambitions swelled, they kept piling on "hacks that bypass the original abstractions," and after a decade of organic evolution they "turned into a giant pile of crap—including Databricks' own." Very few companies have the courage to rewrite from scratch.

To avoid the notorious "second-system syndrome," the team didn't build a new engine by cobbling together the latest academic papers—instead, they built a "database factory." This factory leverages a decade of query traces accumulated by Databricks—on the scale of quadrillions of data points—to train a machine learning model that can predict, with high fidelity, the performance of any algorithm and implementation on any type of query. Based on this, the system can dynamically select the optimal algorithm and data structure at both runtime and implementation time.

Reynold gives a subtle example: whether a string is pure ASCII or contains Unicode has a profound impact on performance. If a set of strings is dense enough (say, only 256 country codes), you can completely replace a hash table with an array lookup—these are exactly the kinds of "counterintuitive" optimizations that the model's millions of features aim to capture. Dream Engine will roll out incrementally, with the new engine required from day one to fully cover existing capabilities and be superior overall (especially for tens-of-milliseconds low-latency workloads), rather than being delivered all at once after years of waiting.

Strategic Foundation: Open-Source DNA and AI Discipline

When pressed on "why they surpassed Snowflake in the compute-storage separation race," Reynold offered two fundamental differences.

The first is open-source DNA. "Databricks never had a proprietary format; we started from an open ecosystem—from Parquet to Delta to Iceberg." He specifically noted that a Snowflake co-founder once wrote an article titled Choosing Open Wisely arguing against open data formats, whereas today open formats have become industry consensus. This trend is a decisive factor for large enterprises that have thirty-year-old legacy systems and have suffered the pain of Oracle lock-in in choosing Databricks.

The second is AI discipline. Even before ChatGPT appeared, Databricks had always positioned itself as a "machine learning + data" platform rather than a pure data infrastructure company. Matei also clarified the model strategy following the acquisition of Mosaic: Databricks did not chase general-purpose frontier large models, but focused on "how to make smart models truly useful"—for example, building the virtual data scientist Genie, and training specialized small models (such as a document-parsing vision model that is about 100x cheaper than frontier models while delivering better results).

---

The two co-founders' conclusions ultimately return to that core proposition: The reasoning capabilities of general-purpose Agents are already good enough; the deciding factor is whether the data is in place. "A lot of traditional software will be rewritten with this new paradigm—get the data in place, connect an Agent, and the magic appears. But without the right data, you can't do anything."

This is precisely the strategic logic behind Databricks building its new products—such as security and customer data platforms—all on top of a data foundation: not AI first, but data first.