Unit Testing DLT Pipelines

dm7 — Mon, 24 Jun 2024 09:38:11 GMT

Now we are moving our DLT Pipelines into production, we would like to start looking at unit testing the transformation logic inside DLT notebooks.

We want to know how we can unit test the PySpark logic/transformations independently without having to spin up a DLT pipeline. Mainly because you can run a DLT notebook and it will output saying it's fine and to create a pipeline, but when you run the pipeline it will then throw the actual errors associated with things like incorrect schema locations etc. It's also hard to debug transformations within DLT as you can't readily inspect inputs/outputs or add debug logic.

Does anyone have any guidance on suitable approaches towards unit testing DLT pipeline notebooks? Thanks

Re: Unit Testing DLT Pipelines

dm7 — Thu, 27 Jun 2024 17:01:51 GMT

Hi Kaniz - what if we have some CDC change data capture stages in a DLT pipeline?
E.g. we have a CDC stage which uses SCD type 1 to take the latest record based on datetime. - How would we go about unit testing this code functions correctly? As it is a native DLT function so couldn't lift and shift this to a separate Python notebook

topic Unit Testing DLT Pipelines in Data Engineering

Unit Testing DLT Pipelines

Re: Unit Testing DLT Pipelines