cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Unit Testing DLT Pipelines

dm7
New Contributor II

Now we are moving our DLT Pipelines into production, we would like to start looking at unit testing the transformation logic inside DLT notebooks.

We want to know how we can unit test the PySpark logic/transformations independently without having to spin up a DLT pipeline. Mainly because you can run a DLT notebook and it will output saying it's fine and to create a pipeline, but when you run the pipeline it will then throw the actual errors associated with things like incorrect schema locations etc. It's also hard to debug transformations within DLT as you can't readily inspect inputs/outputs or add debug logic.

Does anyone have any guidance on suitable approaches towards unit testing DLT pipeline notebooks? Thanks

1 REPLY 1

dm7
New Contributor II

Hi Kaniz - what if we have some CDC change data capture stages in a DLT pipeline?
E.g. we have a CDC stage which uses SCD type 1 to take the latest record based on datetime. - How would we go about unit testing this code functions correctly? As it is a native DLT function so couldn't lift and shift this to a separate Python notebook

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group