Atlassian Teams Up with Databricks: How an OCSF Security Lakehouse Cuts Through 20 Years of Log Noise

Atlassian's security team was long plagued by a seemingly simple problem: how to precisely surface real threat signals from the massive volume of logs generated every day. Two decades of accumulated log data, globally distributed cloud infrastructure, and ever-expanding security alerts — together, these factors overwhelmed traditional SIEM (Security Information and Event Management) architectures.

Their answer was to partner with Databricks to build a security lakehouse based on OCSF (Open Cybersecurity Schema Framework). This system not only reduced noise but also redefined how security data is processed.

Why Traditional Security Architectures Can No Longer Keep Up

Atlassian's business spans the globe, with cloud environments across multiple regions and platforms. The security team needs to collect logs from hundreds of data sources, including cloud service audit logs, network traffic, endpoint detection and response (EDR) alerts, and audit events from in-house applications.

The problem is that these data sources have different formats. Every vendor has its own field naming and log structure, forcing security analysts to spend enormous time on field mapping and normalization rather than focusing on threat hunting.

Cost is even trickier. Traditional SIEMs typically charge by data ingestion volume, and Atlassian's log volume is measured in petabytes. If all raw logs were fed into the SIEM, costs would quickly spiral out of control. If only part of the data were retained, critical evidence could be missed.

Atlassian's engineering team realized they needed an architecture that could retain the full dataset while still enabling low-cost querying. That is how the concept of a security lakehouse came into view.

SIEM (Security Information and Event Management) is a class of platform that centrally collects, correlates, and analyzes security logs, and has long been regarded as the core tool of an enterprise Security Operations Center (SOC). Traditional SIEMs are designed with an emphasis on structured event collection and real-time rule matching, and their pricing models are usually tied to data ingestion volume or peak throughput. This model was relatively economical in an era of low log volume and limited data sources, but when enterprises entered the multi-cloud, containerized, and microservices stage, log types and volumes grew exponentially, and volume-based pricing rapidly amplified costs. At the same time, the storage layer of traditional SIEMs is often coupled with the compute layer, making it hard to scale independently, which forces enterprises to compromise between retention period and query capability. This is exactly the dilemma Atlassian faced: it is not that SIEM is useless, but that its economic model and architectural assumptions no longer fit a petabyte-scale, multi-source, heterogeneous security data environment.

OCSF: Making Security Data Speak the Same Language

OCSF was jointly initiated by vendors including AWS, Splunk, and CrowdStrike, with the goal of defining an open, extensible security event schema. It abstracts security events into several core categories: authentication, network activity, process activity, file activity, vulnerability findings, and more. Each category has standardized field definitions.

This means that no matter which cloud provider or security tool a log comes from, it can be mapped to the same field system. For Atlassian, OCSF solved a data normalization problem that had plagued them for years.

Specifically, they added a transformation layer in the data ingestion pipeline to parse raw logs and map them into OCSF format, then write them into Databricks Delta Lake tables. When security analysts query with SQL or Spark, they face a unified, schema-bearing table rather than dozens of raw logs in different formats.

The efficiency gains from this shift are significant. Queries that previously required custom parsing scripts can now be completed directly using standard fields.

The fundamental reason security data normalization is difficult is that security telemetry is inherently heterogeneous. Cloud provider audit logs, EDR process events, network device traffic records, and in-house application business logs not only differ in field naming, but also in event time granularity, subject identification methods, and semantic boundaries. In the past, enterprises usually wrote parsing rules separately for each data source, and these rules were scattered across different teams and scripts, making them hard to reuse and maintain. The value of OCSF is that it provides a vendor-neutral semantic model: events are grouped into a limited set of categories, fields have clear definitions, and extensions follow specifications. As a result, parsing rules can shift from "targeting a specific vendor's logs" to "mapping to a specific OCSF category," greatly reducing engineering costs when adding new data sources and making cross-source correlation analysis possible at the schema level.

How the Databricks Lakehouse Supports Security Analytics

In choosing Databricks, Atlassian valued three capabilities: open storage, elastic compute, and unified analytics.

As an open table format, Delta Lake stores data on cloud object storage, avoiding vendor lock-in. The security team can retain the full set of raw logs while ensuring data consistency through time travel and ACID transactions.

On the compute side, Databricks' elastic clusters allow Atlassian to dynamically adjust resources based on query load. Daily alert queries use small clusters, while month-end compliance audits or large-scale threat hunting temporarily scale out to much larger capacity. This pay-as-you-go model is more flexible than fixed-capacity SIEMs.

Unified analytics is reflected in the fact that security teams and data engineering teams can collaborate on the same platform. Security analysts use SQL for threat hunting, data engineers use Spark for ETL, and machine learning teams use MLflow to train anomaly detection models — all work is based on the same dataset.

According to Atlassian, after migrating to the lakehouse architecture, security data query performance improved several times over, while storage costs dropped significantly. Although specific numbers were not disclosed, the team said they "no longer need to make painful trade-offs between data retention and cost."

A lakehouse is an architecture that combines the open storage of a data lake with the transactional capabilities of a data warehouse. It is usually based on open table formats such as Delta Lake and Apache Iceberg, providing ACID transactions, schema evolution, and time travel on top of cloud object storage. For security scenarios, this means enterprises can retain the full set of raw logs in low-cost object storage while using SQL or Spark for high-performance queries, without having to choose between "full retention" and "fast querying." The separation of storage and compute also brings elasticity: daily queries use small clusters, while large-scale threat hunting or compliance audits temporarily scale compute resources, with costs tied to actual usage. This model stands in sharp contrast to the fixed-capacity licensing of SIEMs, and is especially suitable for security operations scenarios with highly variable log volumes and diverse query patterns.

The Key to Noise Reduction: From Alert Flood to Signal Filtering

The real pain point for security teams is not data volume, but signal-to-noise ratio. Among the security alerts Atlassian receives every day, a large number are false positives or low-priority events. Analyst time is consumed by repetitive triage work.

The OCSF lakehouse provides the foundation for noise reduction. Because all events are in the same table, the team can write more complex correlation rules. For example, correlate anomalous login events in cloud audit logs with process behavior in EDR, and generate a high-priority alert only when both are triggered at the same time.

In addition, the availability of historical data makes baseline analysis possible. The team can calculate the normal behavior patterns of a user or service, then automatically flag activity that deviates from the baseline. This kind of detection based on historical context is far more accurate than simple threshold alerts.

Atlassian also uses Databricks' machine learning capabilities to cluster and prioritize alerts. Similar alerts are merged, low-risk events are automatically downgraded, and analysts only look at the portion most worthy of attention.

Common techniques for security alert noise reduction include correlation rules, baseline modeling, and alert clustering. Correlation rules filter out false positives from a single source by associating events from multiple data sources by time, subject, or session; for example, an anomalous login alone may just mean the user is traveling, but if the same subject subsequently shows anomalous process behavior, threat confidence rises significantly. Baseline analysis relies on historical data to establish a profile of normal behavior, then uses statistical or machine learning methods to detect deviations. Alert clustering usually uses similarity algorithms to merge large numbers of duplicate or near-duplicate alerts into a single event and rank them by risk score. These methods all require a prerequisite: historical data must be queryable, and multi-source data must be correlatable. Traditional SIEMs, limited by storage costs, often retain only short-term hot data, making it difficult to run baseline analysis and cross-source correlation over the long term, and the lakehouse architecture fills exactly this gap.

Implications for Security Data Architecture

Atlassian's practice reflects a trend in security data analytics: migration from dedicated SIEMs to open lakehouses.

As an open schema, OCSF is gaining support from more and more vendors. It frees security data from the constraints of specific platforms and allows it to flow between different tools. For enterprises with multi-cloud environments like Atlassian, this interoperability is especially important.

The lakehouse architecture, meanwhile, resolves the contradiction between cost and retention for security data. By separating storage and compute, enterprises can retain the full dataset in low-cost object storage and spin up compute resources only when needed.

Of course, this architecture is not without challenges. OCSF mapping requires upfront investment, and maintaining data pipelines also requires specialized engineering capabilities. For enterprises with smaller security teams, adopting a managed SIEM may still be the more pragmatic choice.

But for enterprises with enormous log volumes, complex multi-cloud environments, and data engineering capabilities, Atlassian's path provides a reference model worth considering: use open schemas to unify data, use the lakehouse to reduce costs and break down silos, and ultimately let security analysts return to analysis itself rather than being trapped in data cleansing.

One-line Summary

Atlassian partnered with Databricks to build an OCSF security lakehouse that unifies multi-source logs and significantly lowers security data costs.

Paragraph Summary

Atlassian's security team was long trapped by SIEM cost and noise problems caused by petabyte-scale, multi-source, heterogeneous logs. Together with Databricks, they built a security lakehouse based on the OCSF open schema: in the ingestion pipeline, cloud audit, EDR, network traffic, and other logs are uniformly mapped into OCSF format and written to Delta Lake, enabling full retention and low-cost querying. The lakehouse's separation of storage and compute improved query performance several times over and greatly reduced storage costs. For noise reduction, the unified schema makes cross-source correlation, historical baseline analysis, and machine learning alert clustering possible, freeing analysts from repetitive triage to focus on high-priority signals. The article argues that this practice reflects the industry trend of security data analytics migrating from dedicated SIEMs to open lakehouses, but also cautions that OCSF mapping and data pipeline maintenance require upfront investment and may not be suitable for smaller teams.