Published May 23, 2023
This blog aims to provide a guide on implementing the Elementary package with dbt. The primary functionalities of this guide focus on:
- Integrating Elementary with the jaffle_shop dbt project
- Generating reports on datamart health.
What is Elementary?
Elementary is an open-source data observability package designed for data and analytics engineers using dbt. It provides a comprehensive solution for monitoring and ensuring the quality of your dbt projects and data. With Elementary, you can generate data observability reports, detect data anomalies through dbt tests, analyze models' performance, track dbt artifacts and run results, receive Slack alerts for data issues, and explore data lineage to understand the impact and root causes of data problems. By seamlessly integrating with dbt and leveraging its powerful features, Elementary empowers you to gain immediate visibility, take proactive actions, and maintain the integrity of your data pipelines.
Elementary's functionality revolves around its dbt package, which enables data monitoring and collection of dbt artifacts. Configuration for monitoring is done within your dbt project, using dbt macros and models as monitors. The collected data is stored in a dedicated schema called "elementary" in your data warehouse. To generate UI reports and send Slack alerts, Elementary CLI comes into play. This command-line interface simplifies the process of creating reports and ensures timely notifications of data issues. With Elementary, you can effortlessly monitor your dbt projects, detect anomalies, and gain valuable insights into your data's health and performance, enabling you to proactively address issues and optimize your data workflows.
How to Setup
This guide provides instructions for setting up Elementary using dbt's jaffle_shop as an example project.
- Create a
packages.ymlfile in your dbt project directory if it doesn't already exist. Add the following content to include the Elementary package. Depending on your dbt version, you may need to specify a particular version of dbt utils. Refer to the Elementary documentation for version compatibility details.
- package: elementary-data/elementary
- Install the Elementary package by running the following command:
- Open your
dbt_project.ymlfile and make the following additions to include Elementary:
# Include the following line if you only want Elementary to run when in a specified environment
- Generate the Elementary package models by running the command:
dbt run --select elementary
- This will create a schema named 'Elementary' within the database configured in your
dbt_project.yml, along with tables for models used with the Elementary package for data monitoring.
- Generate your connection profile to allow for the Elementary CLI tool to connect to your database
- This will print out a connection profile that you can add to your existing profiles.yml
dbt run-operation elementary.generate_elementary_cli_profile
- Install the Elementary CLI tool
pip install elementary-data
- Run whatever dbt models you would like
- Run the following command to generate an html report to view your datamart health!