Any way to ignore DLT tables in pipeline

eballinger
Contributor

Hello,

In our testing environment we would like to be able to only update the DLT tables we are testing for our pipeline. This would help speed up the testing. We currently have the pipeline code being generated dynamically based on how many tables there are to be processed. 

What I have discovered though is when I run the pipeline without referencing the table, the DLT table gets removed/deleted. Is there any way to declare a DLT table so that it cannot be removed? 

Just so im clear here is a example:

Pretend there at 10 DLT tables in total for our pipeline and it takes 10 minutes to run.

If I only want to update 2 tables in testing I would like to run the pipeline for those 2 tables only and just ignore the others since they are not being updated. This would then take about 2 minutes.   

I know in the pipeline GUI interface there is a button called "Select tables for refresh" and this does exactly what I want, the only difference is I want to do this functionality in the Python code instead since that is where I dynamically declare the DLT tables.