ETL or Transformations Testing Framework for Databricks

[alexu4798644233]

Hi! I'm looking for any ETL or Transformations Testing Framework for Databricks -
need to support automation of the following steps:

1) create/store test datasets (mock inputs and a golden copy of the output),
2) run ETL (notebook) being tested
3) compare ETL output with the golden copy

Any examples or ready to use options out there?

[Rjdudley]

You can do all of this yourself with a testing workflow.  You can create your data in a notebook or keeping a backup copy of tables, and copy them fresh for your tests.  This would be the first step of the workflow.  Then call your notebooks.  Your comparison can be simple SQL statements which write mis-matches to testing failures tables which you can review.

If you do want a framework, have a look at GitHub - microsoft/nutter: Testing framework for Databricks notebooks.