Databricks Tech Leaders: Why the Frontier AI Ecosystem Must Be Open

Introduction: A Rare Dual Conversation

Two core technical leaders at Databricks — co-founder and CTO Matei Zaharia and co-founder Reynold Xin — sat down for a rare joint interview to explore in depth why the frontier AI ecosystem must remain open, and what key conditions every enterprise needs to build an "Agent Cloud."

This conversation touches on one of the most central debates in the AI industry today: as competition among large models intensifies, how will the battle between open and closed ecosystems shape the future of the entire industry?

The Open AI Ecosystem: More Than Just Idealism

From Spark to AI: Open-Source DNA

As the creator of Apache Spark, Matei Zaharia has a profound understanding of open-source ecosystems. Apache Spark is an open-source distributed computing framework created by Zaharia at UC Berkeley's AMPLab in 2009. Spark's core innovation was the introduction of Resilient Distributed Datasets (RDDs), which keep data in memory for iterative computation — delivering speeds tens to hundreds of times faster than the then-dominant Hadoop MapReduce framework. Spark quickly became the de facto standard for big data processing, adopted by thousands of enterprises worldwide. In 2013, Zaharia and six other co-founders (including Reynold Xin) founded Databricks to commercialize Spark and build an enterprise-grade data and AI platform on top of it. Reynold Xin was primarily responsible for developing the Spark SQL module, which enabled Spark to handle structured data queries and dramatically expanded its range of applications.

Databricks was built on open source from day one, and this DNA has profoundly shaped their perspective on the AI ecosystem. In their view, openness in the frontier AI ecosystem is not merely a technical ideal — it's a practical necessity for industry development.

The AI field is currently at a critical inflection point: shifting from a pure model race to building complete application ecosystems. In this process, enterprises need more than just a powerful foundation model — they need an entire infrastructure stack encompassing data management, fine-tuning, deployment, monitoring, and more. A closed ecosystem means being locked into a single vendor's technology stack, which is an unacceptable risk for most enterprises.

Why the Closed Approach Doesn't Work

From a practical business perspective, closed ecosystems face several fundamental challenges:

• Data sovereignty: An enterprise's core data is its most sensitive asset. Fully hosting it on a closed platform creates significant compliance and security risks.
• Vendor lock-in: AI technology iterates extremely fast. Being locked into one vendor's stack means losing the ability to switch flexibly.
• Customization needs: Every enterprise has different business scenarios. Open ecosystems allow deep customization, while closed platforms typically offer only standardized solutions.

Agent Cloud: The Next Form of Enterprise AI Agents

What Is Agent Cloud?

The "Agent Cloud" concept proposed by Databricks refers to a cloud infrastructure that enables enterprises to build, deploy, and manage AI agents at scale. This isn't simply about calling an API — it's a complete platform covering the entire pipeline: data preparation, model training and fine-tuning, agent orchestration, runtime management, and performance evaluation.

AI agents are fundamentally different from traditional LLM API calls. Traditional calls follow a stateless, single-turn request-response pattern, whereas agents need to autonomously plan tasks, invoke external tools, maintain contextual memory, and make dynamic decisions during multi-step execution. This introduces a series of engineering challenges: agent orchestration requires managing complex workflow graphs, including conditional branches, parallel execution, error retries, and human-in-the-loop nodes; runtime management must handle long-running stateful sessions, which fundamentally conflicts with traditional stateless microservice architectures; additionally, communication protocols, resource scheduling, and conflict resolution in multi-agent collaboration scenarios are pressing problems that need solving. The industry has already seen multiple agent frameworks emerge (such as LangChain, AutoGen, CrewAI, etc.), but enterprise-grade deployment still faces significant challenges in reliability, observability, and scalability. Databricks' Agent Cloud concept aims to systematically address these issues at the platform level.

The core of this vision is that every enterprise should have the capability to build its own AI agent system, rather than relying on generic solutions from a handful of tech giants.

Key Elements for Building an Agent Cloud

To realize this vision, Zaharia and Xin believe several key conditions are necessary:

First, unified data infrastructure. An agent's capabilities largely depend on the quality of data it can access and understand. Databricks has been driving the Lakehouse architecture, unifying the capabilities of data warehouses and data lakes, providing a solid data foundation for AI agents.

The Lakehouse is an architectural paradigm formally introduced by Databricks in 2020, designed to combine the strengths of two traditional architectures: data lakes and data warehouses. Traditional data warehouses (such as Teradata and Snowflake) excel at high-performance queries and transaction processing for structured data, but have high storage costs and limited support for unstructured data. Data lakes (such as those based on HDFS or S3) are inexpensive and can store data in any format, but lack transactional consistency and efficient query capabilities. The Lakehouse architecture introduces open table formats like Delta Lake on top of low-cost object storage, enabling ACID transaction support, schema evolution, time travel, and other enterprise-grade data management capabilities. The key significance of this architecture is that AI and machine learning workloads need simultaneous access to both structured and unstructured data. The Lakehouse eliminates the complexity of moving data between two separate systems, providing AI agents with a unified data access layer.

Second, an open model layer. Enterprises need the freedom to choose and combine different foundation models, including both open-source and commercial models. Databricks' investment in open-source models (such as DBRX) is based precisely on this conviction.

DBRX is an open-source large language model released by Databricks in March 2024, built on a Mixture of Experts (MoE) architecture. The core idea of MoE is to divide model parameters into multiple "expert" sub-networks, activating only a subset during each inference pass. This dramatically reduces actual computational overhead (only about 36B parameters are activated per inference) while maintaining a large total parameter count (DBRX has 132B total parameters). At the time of its release, DBRX outperformed contemporaneous open-source models such as Llama 2 70B and Mixtral on multiple benchmarks. Databricks' strategic intent in developing open-source models is crystal clear: as a platform company, it doesn't need to profit from the model itself. Instead, it enhances its platform's attractiveness by offering high-quality open-source models while reducing enterprises' dependence on any single model provider. This logic is similar to Meta's open-sourcing of the Llama series — expanding ecosystem influence through open-source models and shifting the competitive focus from the model itself to the platform and application layers.

Third, a robust governance and security framework. When AI agents begin executing business operations on behalf of humans, permission management, audit trails, and security controls become critically important.

Industry Landscape and Competitive Analysis

The Open vs. Closed Battle

The AI infrastructure market is currently splitting into two camps. On one side is the more closed approach represented by OpenAI, emphasizing end-to-end integrated experiences. On the other is the open approach represented by Meta (Llama series), Databricks, and others, emphasizing ecosystem diversity and enterprise autonomy.

Zaharia and Xin's position is clear: in the enterprise market, the open approach will ultimately prevail. History has proven this repeatedly — from Linux to Kubernetes, from Hadoop to Spark, the winners in enterprise infrastructure have almost always been open ecosystems.

These cases reveal a common historical pattern. In the 1990s, Linux gradually displaced various commercial Unix variants (such as Solaris, AIX, and HP-UX) in the server operating system market, eventually capturing over 90% of global server market share. In the 2000s, the Hadoop ecosystem replaced expensive commercial data processing solutions, although Hadoop itself was later superseded by more advanced open-source technologies like Spark. In the 2010s, Kubernetes evolved from Google's internal Borg system and was open-sourced, quickly defeating competing solutions like Docker Swarm and Mesos to become the de facto standard for container orchestration. These cases reveal a common pattern: in enterprise infrastructure, open standards and open-source technologies ultimately win because enterprises need to avoid vendor lock-in, need community-driven rapid innovation, and need consistency across multi-cloud and hybrid environments. As the next generation of core enterprise infrastructure, AI infrastructure will most likely follow this same pattern.

Implications for China's AI Ecosystem

This discussion holds equally important lessons for China's AI industry. Domestic AI companies face the same open-versus-closed choice when building their own agent platforms. In the long run, building an open, interoperable AI infrastructure ecosystem may be more sustainable than creating a closed, all-in-one suite.

Summary and Outlook

The core argument from Databricks' two technical leaders can be distilled into a single statement: The future of AI belongs to open ecosystems, because only openness can empower every enterprise to be a builder of AI, not merely a consumer.

As AI agents move from concept to large-scale deployment, the infrastructure competition around Agent Cloud will become the primary battleground in the next phase. In this competition, whoever can provide the most open, most flexible, and most user-friendly platform stands to become the infrastructure winner of the enterprise AI era.

Key Takeaways