---
title: Streamline your Data Transformations by Running dbt Directly on Databricks using Jobs
description: Running dbt (data build tool) on Databricks is a great alternative to dbt Cloud if you are looking to automate your data transformation workflows directly on Databricks. In this blog, I will explain step by step how easy it is to set up a Databricks job that transforms your data
image: https://blog.ippon.tech/hubfs/Imported_Blog_Media/streamline-your-data-transformations-by-running-dbt-directly-on-databricks-using-jobs-3.png
---
[Skip to main content](https://blog.ippon.tech/streamline-your-data-transformations-by-running-dbt-directly-on-databricks-using-jobs#main)
[   ](https://ipponusa.com/)
- [Home](https://ipponusa.com)
- [Show submenu for About About](https://ipponusa.com/about-us)
- [Who We Are](https://ipponusa.com/who-we-are)
- [Careers](https://ipponusa.com/careers)
- [Show submenu for Our Partners Our Partners](https://ipponusa.com/our-partners/)
- [AWS](https://ipponusa.com/our-partners/aws/)
- [Snowflake](https://ipponusa.com/our-partners/snowflake/)
- [Databricks](https://ipponusa.com/our-partners/databricks/)
- [Microsoft](https://ipponusa.com/our-partners/microsoft/)
- [Show submenu for Services Services](https://ipponusa.com/services)
- [Snowflake](https://ipponusa.com/service/snowflake-concierge/)
- [Artificial Intelligence](https://ipponusa.com/service/artificial-intelligence/)
- [Data & Analytics](https://ipponusa.com/service/data-analytics/)
- [Cloud Strategy](https://ipponusa.com/service/cloud-strategy/)
- [Platform Modernization](https://ipponusa.com/service/platform-moderization/)
- [Product Innovation](https://ipponusa.com/service/product-innovation/)
- [Operating Model](https://ipponusa.com/service/operating-model/)
- [Blogs](https://blog.ippon.tech/)
- [Success Stories](https://ipponusa.com/success-stories/)
- [Show submenu for Resources Resources](https://ipponusa.com/resources/)
- [eBooks](https://ipponusa.com/ebooks/)
- [The Data Pour](https://info.ippon.tech/the-data-pour)
- [Videos & Webinars](https://info.ippon.tech/videos-and-webinars)
- [Media Center](https://ipponusa.com/media-center/)
Search
Open main navigation
Close main navigation
- [Home](https://ipponusa.com)
- Show submenu for About About
- About
- [About](https://ipponusa.com/about-us)
- [Who We Are](https://ipponusa.com/who-we-are)
- [Careers](https://ipponusa.com/careers)
- Show submenu for Our Partners Our Partners
- Our Partners
- [Our Partners](https://ipponusa.com/our-partners/)
- [AWS](https://ipponusa.com/our-partners/aws/)
- [Snowflake](https://ipponusa.com/our-partners/snowflake/)
- [Databricks](https://ipponusa.com/our-partners/databricks/)
- [Microsoft](https://ipponusa.com/our-partners/microsoft/)
- Show submenu for Services Services
- Services
- [Services](https://ipponusa.com/services)
- [Snowflake](https://ipponusa.com/service/snowflake-concierge/)
- [Artificial Intelligence](https://ipponusa.com/service/artificial-intelligence/)
- [Data & Analytics](https://ipponusa.com/service/data-analytics/)
- [Cloud Strategy](https://ipponusa.com/service/cloud-strategy/)
- [Platform Modernization](https://ipponusa.com/service/platform-moderization/)
- [Product Innovation](https://ipponusa.com/service/product-innovation/)
- [Operating Model](https://ipponusa.com/service/operating-model/)
- [Blogs](https://blog.ippon.tech/)
- [Success Stories](https://ipponusa.com/success-stories/)
- Show submenu for Resources Resources
- Resources
- [Resources](https://ipponusa.com/resources/)
- [eBooks](https://ipponusa.com/ebooks/)
- [The Data Pour](https://info.ippon.tech/the-data-pour)
- [Videos & Webinars](https://info.ippon.tech/videos-and-webinars)
- [Media Center](https://ipponusa.com/media-center/)
- Search
- [Contact Us](https://ipponusa.com/contact/)
[Contact Us](https://ipponusa.com/contact/)
# Streamline your Data Transformations by Running dbt Directly on Databricks using Jobs
by [Theo LEBRUN](https://blog.ippon.tech/author/theo-lebrun)
April 26, 2023
[Data](https://blog.ippon.tech/tag/data/) [Databricks](https://blog.ippon.tech/tag/databricks/) [DBT](https://blog.ippon.tech/tag/dbt/) [ETL](https://blog.ippon.tech/tag/etl/)
*Published April 2023*
Running [dbt (data build tool)](https://www.getdbt.com/?ref=blog.ippon.tech) on Databricks is a great alternative to dbt Cloud if you are looking to automate your data transformation workflows directly on Databricks. In this blog, I will explain step by step how easy it is to set up a Databricks job that transforms your data using dbt.
## Transform data with dbt
If you never used dbt before, you will first need to be more familiar with a few concepts like models and macros. I recommend reading the [documentation](https://docs.getdbt.com/docs/introduction?ref=blog.ippon.tech) for more details.
We first need a dataset, as a New Yorker I decided to pick a very controversial subject which is ... Pizzas! Our friend [Liam Quigley](https://elkue.com/nyc-slice/?ref=blog.ippon.tech) went on a journey and ate 464 slices from multiple pizza places in NYC while logging everything. He also created a [spreadsheet](https://docs.google.com/spreadsheets/d/1EY3oi9ttxybG0A0Obtwey6BFu7QLqdHe02JApijgztg/edit?ref=blog.ippon.tech#gid=0) and for this blog I created a table in Databricks with the data:
The second step is to create a dbt project that will transform our pizza raw table and create tables/views. I will use this [dbt project on Github](https://github.com/Falydoor/blog-databricks-dbt?ref=blog.ippon.tech) that has 3 models:
- pizza: A table that parses the date and add 2 columns (id and inserted\_at)
- name\_summary: A view that counts pizza places by name
- style\_summary: A view that calculate the total spent by pizza's style
## Why dbt and Databricks
I am a huge fan of using dbt with Databricks and it is very easy to have them running together thanks to the [Databricks adapter](https://github.com/databricks/dbt-databricks?ref=blog.ippon.tech). All models use Delta Lake out of the box and you also have a few extra features like [snapshots](https://docs.getdbt.com/docs/build/snapshots?ref=blog.ippon.tech) and [merge](https://docs.getdbt.com/reference/resource-configs/spark-configs?ref=blog.ippon.tech#the-merge-strategy). Unity Catalog is supported and the adapter also includes SQL macros that are optimized to run with the [Photon runtime](https://docs.databricks.com/runtime/photon.html?ref=blog.ippon.tech).
It's safe to say that Databricks pairs very well with dbt and is also a good alternative to Delta Live Tables for your ETL pipelines. For more details on the configuration and how to use the adapter, please read [this page](https://docs.getdbt.com/reference/resource-configs/spark-configs?ref=blog.ippon.tech).
## Databricks Workflows
Databricks Workflows is a fully-managed orchestration service that is highly reliable and very easy to use. We're going to use that to create and run our dbt job directly on Databricks without configuring any external resources.
Before creating the dbt job, make sure that Databricks SQL is enabled for your workspace and that you have a SQL warehouse created. Also, your dbt project must be a Databricks Repos, it can't run directly from DBFS.
You can create a job in the Databricks UI or using the Databricks API. After creating the job, add a new task that uses the type `dbt` and link the source to your dbt project:
The `dbt commands` is very important and defines the order of the dbt commands run by the task. Feel free to specify any flags like `--full-refresh` for a full refresh or `-m my_model` for a specific model. Don't forget to set the correct `schema` and also the right cluster to run the dbt CLI on.
## Run and monitor job
You can configure the job to run on a schedule or trigger it manually. After triggering a run, you can monitor its progress in the Databricks UI or using the Databricks API. You can view the job run history, logs, and metrics. You can also configure alerts to notify you when the job fails or exceeds a certain threshold.
The output of the Job run displays our dbt logs, and they can be inspected to make sure that everything was completed successfully.
My dbt project is configured to run tests using dbt's generic tests, and we can see that the test `unique_pizza_id` failed:
After a quick SQL query, we can find out what happened:
Looks like there is a mistake in our dataset unless it is normal to pay a different price for the same pizza...
## Conclusion
Running dbt directly on Databricks using a job is a powerful way to automate your data transformation workflows. With Databricks, you can easily set up a cluster, create a dbt project, and schedule the execution of your dbt commands. [Read this article](https://www.databricks.com/blog/2022/12/15/best-practices-super-powering-your-dbt-project-databricks.html?ref=blog.ippon.tech) for best practices and advices on running your dbt project with Databricks.
Need help with your existing Databricks Lakehouse Platform and dbt? Or do you need help using the latest Data lake technology? Ippon can help! Send us a line at [contact@ippon.tech](mailto:contact@ippon.tech).
## Related Articles
##### [ Data • April 16, 2026 RVATechData + AI Summit: My First Conference Experience 4 min read](https://blog.ippon.tech/rvatechdata-ai-summit-my-first-conference-experience)
##### [ Data • July 1, 2025 The Evolution of AI : Part Two 4 min read](https://blog.ippon.tech/the-evolution-of-ai-part-two)
##### [ Data • June 25, 2025 Data Management in the Age of Generative AI 7 min read](https://blog.ippon.tech/data-management-in-the-age-of-generative-ai)
### Comments
### Subscribe to Our Blog!
Stay informed with the latest insights and updates by signing up for our weekly blog newsletter – delivered straight to your inbox!
[Back to blog homepage »](https://blog.ippon.tech/)
Ippon is a consulting and expertise firm, who is convinced that technology is a source of progress for society. We help our clients leverage their digital assets to design an appropriate strategy and deploy their transformation roadmap at scale.
#### Navigation
- [Home](https://ipponusa.com/)
- [About](https://ipponusa.com/about-us/)
- [Services](https://ipponusa.com/services/)
- [Blogs](https://blog.ippon.tech/?__hstc=223043268.1036d82247623b329dfefe4e92697801.1714729179243.1714729179243.1714729179243.1&__hssc=223043268.2.1714729179244&__hsfp=803678701)
- [eBooks](https://ipponusa.com/ebooks/)
- [The Data Pour](https://info.ippon.tech/the-data-pour?__hstc=223043268.1036d82247623b329dfefe4e92697801.1714729179243.1714729179243.1714729179243.1&__hssc=223043268.2.1714729179244&__hsfp=803678701)
[Contact Us](https://ipponusa.com/contact/)
[Join Us](https://ipponusa.com/careers/)
#### Services
- [Data & Analytics](https://ipponusa.com/service/data-analytics/)
- [Cloud Strategy](https://ipponusa.com/service/cloud-strategy/)
- [Artificial Intelligence](https://ipponusa.com/service/artificial-intelligence/)
- [Operating Model](https://ipponusa.com/service/operating-model/)
- [Platform Modernization](https://ipponusa.com/service/platform-moderization/)
- [Product Innovation](https://ipponusa.com/service/product-innovation/)
#### Ippon International
- [France](https://fr.ippon.tech/?__hstc=223043268.1036d82247623b329dfefe4e92697801.1714729179243.1714729179243.1714729179243.1&__hssc=223043268.2.1714729179244&__hsfp=803678701)
- [Australia](https://au.ippon.tech/?__hstc=223043268.1036d82247623b329dfefe4e92697801.1714729179243.1714729179243.1714729179243.1&__hssc=223043268.2.1714729179244&__hsfp=803678701)
- #### Contact
- Ippon Technologies
[3431 West Leigh Street Richmond, VA 23230, USA](https://maps.app.goo.gl/RySm6SAwZnsxeiLz9)
- [(844) 477- 6687](tel:8444776687)
- [Sales@ipponusa.com](mailto:Sales@ipponusa.com)
©Copyright 2024 Ippon USA. All Rights Reserved. | [Terms and Conditions](https://ipponusa.com/privacy-policy/) | [Privacy Policy](https://ipponusa.com/privacy-policy/) | [Website by Skol Marketing](https://skolmarketing.com/)
```json
{
"@context" : "https://schema.org",
"@type" : "BlogPosting",
"author" : {
"@type" : "Person",
"name" : "Theo LEBRUN",
"url" : "https://blog.ippon.tech/author/theo-lebrun"
},
"dateModified" : "2024-01-29T16:09:43.138Z",
"datePublished" : "2023-04-26T18:05:00.000Z",
"headline" : "Streamline your Data Transformations by Running dbt Directly on Databricks using Jobs",
"image" : [ "https://blog.ippon.tech/hubfs/Imported_Blog_Media/streamline-your-data-transformations-by-running-dbt-directly-on-databricks-using-jobs-3.png" ],
"mainEntityOfPage" : {
"@id" : "https://blog.ippon.tech/streamline-your-data-transformations-by-running-dbt-directly-on-databricks-using-jobs",
"@type" : "WebPage"
},
"publisher" : {
"@type" : "Organization",
"logo" : {
"@type" : "ImageObject",
"url" : "https://blog.ippon.tech/hubfs/logo_SVG.svg"
},
"name" : "Ippon Technologies"
}
}
```