Enterprise AI Governance: How to Get Visibility into AI Usage, Costs, and Outcomes

AI Is Everywhere — But Governance Is a Blind Spot

Generative AI has permeated virtually every aspect of enterprise operations — from code assistance and customer service automation to internal knowledge management. Yet as AI tools spread rapidly across organizations, a critical question emerges: Do enterprises actually know who is using these AI systems, how much they're spending, and whether they're delivering real value?

In a recent conversation shared by Databricks, CTO Matei Zaharia and team member Kuhlenhuth explored this very topic. They argued that the next phase of AI adoption is no longer about "can we use AI" — it's about "can we govern AI effectively." This is precisely why enterprise AI governance has become such a pressing topic.

Three Core Questions in Enterprise AI Governance

As AI applications proliferate inside organizations, leadership and engineering teams face unprecedented visibility challenges. Zaharia and Kuhlenhuth distilled these into three core questions:

Who Is Using Which AI Tools?

When a company simultaneously connects to OpenAI, Anthropic, open-source models, and various AI SDKs, it often lacks a unified view of which models and tools each team is actually calling. This "shadow AI" phenomenon mirrors the early days of "shadow IT" — tools are adopted organically by business teams, operating outside centralized management and creating security and compliance risks.

The concept of "shadow AI" is borrowed from "shadow IT" — referring to employees purchasing or using software tools without formal IT approval. In the generative AI era, this has spread far faster than before: developers only need an API key to access GPT-4 or Claude, and business teams can access AI features simply by subscribing to SaaS products — all without going through security reviews, data compliance assessments, or procurement processes. The risks include sensitive enterprise data being sent to unvetted third-party models, duplicate payments across teams for the same functionality, and an inability to trace accountability when security incidents occur. For industries subject to GDPR, HIPAA, or financial regulations, this invisibility can constitute direct compliance violations.

What Is AI Actually Costing?

The cost structure of AI calls is complex and difficult to predict. Token pricing varies enormously across models, and a seemingly simple batch task can cost several times more simply because the wrong model was chosen. Without granular cost tracking, organizations can easily face surprise bills at the end of the month. Clear cost data is a prerequisite for optimizing AI return on investment.

Is AI Delivering Quality Results?

This is perhaps the most commonly overlooked — yet most important — question. AI is running, but how good is the output? Is it actually driving business outcomes? Without continuous monitoring of output quality, AI investment risks becoming a "looks busy" tech showcase rather than a genuine value driver.

Unity Gateway: A Unified AI Observability and Governance Layer

To address these pain points, Databricks introduced Unity Gateway — positioned as a unified entry point for enterprise AI governance and observability. Through this gateway, teams can:

• Centrally manage AI access: Consolidate dispersed model and API calls into a single entry point, enabling unified control over permissions, auditing, and compliance;
• Granularly track cost flows: Clearly surface AI spend by team and project, turning costs from a "black box" into a transparent ledger;
• Systematically evaluate output quality: Integrate evaluation mechanisms to help teams determine whether AI is actually meeting its intended goals.

This "gateway + governance" model essentially brings AI applications into an enterprise-grade observability framework — similar to how we've always treated databases, microservices, and cloud resources. As AI becomes part of the infrastructure, it deserves the same level of monitoring and management.

Observability is a concept borrowed from software engineering, originally used to measure whether a system's internal state can be inferred solely from its external outputs. Traditionally, it rests on three pillars: Logs, Metrics, and Traces. Applying this framework to AI systems is challenging because AI "output" isn't just request-response pairs — it also includes the quality of generated content, hallucination rates, latency distributions, and correlation with business outcomes, all of which are far harder to quantify than traditional API calls. By bringing AI calls into an observability framework, Unity Gateway enables organizations to monitor model invocations the way they monitor microservice performance — catching anomalies before they escalate, rather than discovering problems at month-end billing or through user complaints.

Omnigent: Secure Reuse and Sharing of Coding Agents

Beyond the governance layer of Unity Gateway, the team also introduced Omnigent — a collaboration capability aimed at developers. Its core value is enabling builders to securely compose and share existing coding agents and SDKs.

In practice, teams often maintain their own siloed AI coding toolchains, reinventing the wheel and struggling to collaborate. Omnigent aims to solve this fragmentation: developers can safely encapsulate, reuse, and share existing agents and SDKs without worrying about permission leakage or environment contamination. For engineering organizations looking to scale AI-assisted programming, this means greater reuse efficiency and more controllable security boundaries.

A coding agent is an AI system capable of autonomously completing software development tasks such as code generation, debugging, refactoring, or testing. These systems are typically built on large language models combined with tool-use capabilities. Unlike simple code completion, coding agents can execute multi-step tasks — reading codebases, running terminal commands, executing tests, and iterating based on results. Current mainstream coding agents include GitHub Copilot Agent, Cursor, and Devin. In enterprise environments, different teams often build their own specialized agents on top of the same underlying models, resulting in significant duplicated effort. Omnigent directly targets this fragmentation: through secure encapsulation and permission isolation, it allows teams to share validated agent capabilities, reducing redundant development costs while maintaining clear security boundaries.

AI Governance Is Becoming a Core Enterprise Requirement

This conversation reveals a clear trend: the democratization of AI capabilities is shifting from an "adoption phase" to a "governance phase." Early on, enterprises were racing to get AI up and running. Now, visibility, cost control, and quality assurance have become the decisive factors in whether AI investments succeed or fail.

This reflects a maturation of enterprise AI. When AI expands from experiments by a handful of teams to an organization-wide productivity tool, the cost of ungoverned AI is dramatically amplified — uncontrolled spending, unauditable call histories, and inconsistent output quality can all erode the very value AI was supposed to create.

The answer Databricks offers through Unity Gateway and Omnigent represents a "platform-based governance" philosophy: not restricting AI use, but making AI use visible, controllable, and measurable. For enterprises scaling AI into production, this may be exactly where the most important investments lie in the next phase.