cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

ETL or Transformations Testing Framework for Databricks

alexu4798644233
New Contributor II

Hi! I'm looking for any ETL or Transformations Testing Framework for Databricks -
need to support automation of the following steps:

1) create/store test datasets (mock inputs and a golden copy of the output),
2) run ETL (notebook) being tested
3) compare ETL output with the golden copy

Any examples or ready to use options out there?

1 REPLY 1

Rjdudley
Valued Contributor II

You can do all of this yourself with a testing workflow.  You can create your data in a notebook or keeping a backup copy of tables, and copy them fresh for your tests.  This would be the first step of the workflow.  Then call your notebooks.  Your comparison can be simple SQL statements which write mis-matches to testing failures tables which you can review.

If you do want a framework, have a look at GitHub - microsoft/nutter: Testing framework for Databricks notebooks.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group