topic Obtain refresh mode from within Delta Live Table pipeline run in Data Engineering

Obtain refresh mode from within Delta Live Table pipeline run

NemesisMF — Tue, 05 Nov 2024 14:33:04 GMT

Is it possible to obtain somehow if a DLT pipeline run is running in Full Refresh or incremental mode from within a notebook running in the pipeline?I looked into the pipeline configuration variables but could not find anything.

It would be benefitial to have this information from within the code and do something different in case of a full refresh.

My workaround is so far to have two pipeline jobs and set a config variable if it is running in full refresh, but when executing the pipeline manually this gets dangerous since I have to remind myself to the the value to the correct refresh type.

Re: Obtain refresh mode from within Delta Live Table pipeline run

Walter_C — Tue, 05 Nov 2024 16:10:27 GMT

You can use the following code:

pipeline_run_config = get_current_pipeline_run_config() # Create a StartUpdate object from the pipeline run configuration start_update = StartUpdate.from_dict(pipeline_run_config) # Check if the pipeline run is a full refresh if start_update.full_refresh: print("The pipeline is running in Full Refresh mode.") else: print("The pipeline is running in Incremental mode.")

Re: Obtain refresh mode from within Delta Live Table pipeline run

NemesisMF — Tue, 05 Nov 2024 17:05:58 GMT

Thanks for the quick answer. Where did you get the

get_current_pipeline_run_config()

from? I used spark.conf.getAll which apperently does not have the refresh mode info.

StartUpdate

comes from the databricks.sdk.service.pipelines?

Re: Obtain refresh mode from within Delta Live Table pipeline run

Walter_C — Tue, 05 Nov 2024 19:32:42 GMT

I am looking further on this with our teams, can you please provide us with more context on your usecase for this information?

Re: Obtain refresh mode from within Delta Live Table pipeline run

NemesisMF — Wed, 13 Nov 2024 08:02:51 GMT

We found a solution where we do not need to determine the refresh mode anymore. But I still do not know how to get the current refresh mode of the current pipeline run from within a notebook that is running in the pipeline. This may would still be beneficial to others.