cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

DLT pipeline task with full refresh once in a while

Walter_N
New Contributor

Hi all, I'm using Databricks workflow with some dlt pipeline tasks. These tasks requires a full refresh at some times due to schema changes in the source. I've been doing the full refresh manually or set the full refresh option in the job settings, then let it run once and switch off after that. 
Is there any ways that I can manage to run the job with full refresh option just once and turn it off until next full refresh required, either with some defined rules or triggers? I'm using Databricks with AWS and Terraform for resources provision.
Thanks everyone in advance 

1 ACCEPTED SOLUTION

Accepted Solutions

MariuszK
Contributor III

Hi,

Did you check a possibility to use if/else task? You could define some scriteria and pass it from a notebok that will check if it's time for full refresh or just resfres.

View solution in original post

2 REPLIES 2

MariuszK
Contributor III

Hi,

Did you check a possibility to use if/else task? You could define some scriteria and pass it from a notebok that will check if it's time for full refresh or just resfres.

I did try as you suggested: create an if/else, forward to same tasks with full refresh option on/off. This worked well, but something I considered as duplicated tasks creation and state change management for criteria check. I would prefer a neat workaround to easily apply with Terraform for massive jobs provision.
It is more of a code management and future expansion problem. If Databricks has the option for one time full refresh, it would be a relief.
Appreciate your suggestion. I guess this could be handled better on my side for now.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group