A Practical Guide to Migrating dbt ETL Pipelines to Databricks

More and more data teams are moving their dbt transformation workloads to the Databricks Lakehouse platform. This combination — dbt handling declarative data transformation logic while Databricks provides unified lakehouse storage and compute — is quickly becoming one of the mainstream configurations in the modern data stack. This article walks through the key steps and considerations for repointing an existing dbt ETL pipeline to Databricks.

Note: This article is based on limited source material, and some content reflects supplementary interpretation drawn from common industry practices. Always refer to official documentation for specific implementations.

Why Choose the dbt + Databricks Combination

The core value of dbt (data build tool) lies in managing data transformations using SQL and Jinja templates, bringing software engineering best practices — version control, testing, documentation, and modularity — into the data modeling workflow. Databricks Lakehouse, on the other hand, unifies the low-cost storage of a data lake with the query performance of a data warehouse, built on the open Delta Lake format.

When combined, dbt handles the "transformation layer" (the T in ELT), while Databricks provides elastically scalable compute resources and persistent storage. Compared to running dbt on a traditional data warehouse, migrating to Databricks typically delivers better large-scale processing capabilities, more flexible cost control, and native support for semi-structured data.

Delta Lake is an open-source storage layer format created by Databricks, built on top of Parquet files. Its core features include: ACID transaction support (via a transaction log), scalable metadata handling, Time Travel (data version rollback), and Schema Evolution. It addresses the longstanding pain points of traditional data lakes — lack of transactional guarantees and susceptibility to data inconsistencies. In the context of dbt, Delta Lake means that tables written by dbt models natively support merge/upsert operations, incremental model implementations are more reliable than with traditional object storage, and Time Travel enables quick rollbacks to historical versions when data quality issues arise.

Pre-Migration Preparation

Before diving into the "repoint" process, a few foundational tasks need to be in place:

Environment and Connection Configuration

The core change is replacing the adapter and connection configuration in your dbt project. dbt communicates with Databricks via the dbt-databricks adapter. You'll need to configure the connection details for your target cluster or SQL Warehouse in profiles.yml, including the host, HTTP path, access token, catalog, and schema.

Auditing SQL Dialects in Existing Models

SQL dialects vary across data warehouses. When migrating from Snowflake, BigQuery, or Redshift, you'll need to review the functions, data types, and syntax used in your existing models to confirm they have equivalent implementations in Databricks SQL (which is based on Spark SQL). Date functions, window functions, and type casting are the most common areas of divergence.

Core Migration Steps

Installing and Configuring the Adapter

Start by installing the dbt-databricks package and specifying type: databricks in your profile. Compared to the earlier dbt-spark adapter, the official recommendation is to use dbt-databricks, which has fuller support for Unity Catalog and Delta table features.

Unity Catalog is Databricks' unified data governance layer, using a three-level namespace: catalog.schema.table — unlike the traditional two-level namespace (schema.table). Once Unity Catalog is enabled, permission management, data lineage tracking, and audit logging for all data assets are handled at this layer. For dbt users, the most immediate impact is that the catalog field must be specified in profiles.yml, and all model references and paths generated by the ref() function will carry the catalog prefix. Teams previously using the dbt-spark adapter or running in older workspaces without Unity Catalog enabled should pay close attention to namespace changes when migrating to dbt-databricks + Unity Catalog, and update any cross-database model references accordingly.

Adjusting Materialization Strategies

In a Databricks environment, the materialization approach for dbt models needs to be re-evaluated. Delta Lake supports multiple materialization types including table, view, incremental, and materialized view. For incremental models, you can leverage Delta's merge capability to implement efficient upserts — this is often the key performance optimization opportunity after migration.

dbt's incremental model uses the is_incremental() macro to determine whether the current run is a full refresh or an incremental load. In a Databricks environment, configuring incremental_strategy: merge lets dbt generate upsert logic based on Delta Lake's MERGE INTO statement — matching existing rows by a specified unique key and updating them, or inserting new rows when no match is found — avoiding the full rewrite overhead of the traditional insert overwrite approach. Beyond merge, Databricks also supports insert_overwrite (partition-level overwrite) and append strategies. The right choice depends on your trade-offs between idempotency, data latency, and compute cost. For high-frequency, small-batch update scenarios, the merge strategy combined with Delta's Z-Order indexing can significantly reduce scan ranges and further improve performance.

Validating Data Consistency

Once migration is complete, it's essential to compare data between the old and new environments to confirm that transformation results are consistent. dbt's built-in test framework (e.g., unique, not_null, accepted_values) can serve as a baseline validation mechanism. For more complex scenarios, consider writing custom tests or performing row count and aggregate value comparisons.

Common Pitfalls and Optimization Tips

One of the most frequently overlooked aspects of migration is compute resource selection. Databricks offers multiple compute options — interactive clusters, job clusters, and SQL Warehouses. dbt's batch transformation workloads are generally better suited to Serverless SQL Warehouses or job clusters, which balance cost against startup latency.

Permission and Catalog management also need to be planned upfront. If Unity Catalog is enabled, the target catalog/schema for dbt must have the appropriate permissions configured — otherwise model creation will fail.

Additionally, treat the migration as an opportunity for technical debt cleanup: audit redundant models, standardize naming conventions, and fill in missing documentation and tests, rather than mechanically porting over old logic.

Summary

Repointing a dbt pipeline to Databricks is not as simple as swapping a connection string — it's a systematic effort involving adapter configuration, SQL dialect adaptation, materialization strategy adjustments, and data validation. With proper upfront auditing and thorough testing, the dbt + Databricks combination can deliver meaningful improvements in both maintainability and scalability for your data team.