topic Re: DLT pipeline task with full refresh once in a while in Data Engineering

DLT pipeline task with full refresh once in a while

Walter_N — Fri, 14 Mar 2025 08:11:02 GMT

Hi all, I'm using Databricks workflow with some dlt pipeline tasks. These tasks requires a full refresh at some times due to schema changes in the source. I've been doing the full refresh manually or set the full refresh option in the job settings, then let it run once and switch off after that.
Is there any ways that I can manage to run the job with full refresh option just once and turn it off until next full refresh required, either with some defined rules or triggers? I'm using Databricks with AWS and Terraform for resources provision.
Thanks everyone in advance

Re: DLT pipeline task with full refresh once in a while

MariuszK — Fri, 14 Mar 2025 09:41:21 GMT

Hi,

Did you check a possibility to use if/else task? You could define some scriteria and pass it from a notebok that will check if it's time for full refresh or just resfres.

Re: DLT pipeline task with full refresh once in a while

Walter_N — Fri, 14 Mar 2025 10:37:34 GMT

I did try as you suggested: create an if/else, forward to same tasks with full refresh option on/off. This worked well, but something I considered as duplicated tasks creation and state change management for criteria check. I would prefer a neat workaround to easily apply with Terraform for massive jobs provision.
It is more of a code management and future expansion problem. If Databricks has the option for one time full refresh, it would be a relief.
Appreciate your suggestion. I guess this could be handled better on my side for now.