Databricks Community

eballinger

Hello,In our testing environment we would like to be able to only update the DLT tables we are testing for our pipeline. This would help speed up the testing. We currently have the pipeline code being generated dynamically based on how many tables th...

eballinger

Hi All,We have sensitive tables and have applied row level security and column masking. I would like to build into our job a check to make sure these tables still have the row filters and column masks applied. This would help ensure these security fi...

eballinger · 12-02-2024

Hi Guys,I am new to this community. I am guessing we have a typical setup (DLT tables, 3 layers - bronze, silver and gold) and while it works fine in our development environment I have always looked for ways to speed things up for testers. For exampl...

eballinger

Thanks Riz, I have not worked with the API yet but just tried your suggestion and it worked great. I will share my code for someone else looking for this: myList = ["table1","table6"]#Now run the pipelineupdate_dlt2 = w.pipelines.start_update(refresh...

eballinger

Just wondering if there is a way to do this through code / not using the UI. Its now Dec 2024. We would like to handle this in testing using our workflowsThanks

eballinger

Hi again Alberto,I just tested your solution and I think I missed what you were saying about it only persists while the pipeline is being run. That might work for some other scenarios, but in my example case above I want all of my 10 DLT tables to ex...

eballinger · 12-03-2024

Thanks Walter. All good suggestions. We have tried your 1 and 2 suggestions already. The problem is testers need to test different tables so this first strategy has limits. But I will look into your other suggestions as well. I know you can now run t...

Databricks Community

User Stats

User Activity

Any way to ignore DLT tables in pipeline

Check for row level security and column masking

Looking for ways to speed up DLT testing

Re: How to refresh a single table in Delta Live Tables?

Re: How to refresh a single table in Delta Live Tables?

Re: Any way to ignore DLT tables in pipeline

Re: Looking for ways to speed up DLT testing