topic Re: ETL or Transformations Testing Framework for Databricks in Data Engineering

ETL or Transformations Testing Framework for Databricks

alexu4798644233 — Fri, 24 Jan 2025 13:22:11 GMT

Hi! I'm looking for any ETL or Transformations Testing Framework for Databricks -
need to support automation of the following steps:

1) create/store test datasets (mock inputs and a golden copy of the output),
2) run ETL (notebook) being tested
3) compare ETL output with the golden copy

Any examples or ready to use options out there?

Re: ETL or Transformations Testing Framework for Databricks

Rjdudley — Fri, 24 Jan 2025 14:10:11 GMT

You can do all of this yourself with a testing workflow. You can create your data in a notebook or keeping a backup copy of tables, and copy them fresh for your tests. This would be the first step of the workflow. Then call your notebooks. Your comparison can be simple SQL statements which write mis-matches to testing failures tables which you can review.

If you do want a framework, have a look at GitHub - microsoft/nutter: Testing framework for Databricks notebooks.

Re: ETL or Transformations Testing Framework for Databricks

rameshcsert — Tue, 31 Mar 2026 21:47:33 GMT

Hi Rjdudley, tuff for me to understand the readme file and execute the framework. can you post video of how to install and use for any custom data source with customization test cases