Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-25-2025 02:22 AM
Hello,
There are two ways to determine whether a DLT pipeline is running in Full Refresh or Incremental mode:
DLT Event Log Schema
The details column in the DLT event log schema includes information on "full_refresh". You can use this to identify whether it is True or False.DLT Event Log Schema Documentation
An example of the details column is as follows:
{"user_action":{"action":"START","user_name":"xxxxxxx@gmail.com","user_id":xxxxxxxx,"request":{"start_request":{"full_refresh":false,"validate_only":false}}}}Databricks REST API
You can retrieve DLT pipeline information using the Databricks REST API, which also contains the "full_refresh" field. Here, you can check whether it is True or False.Since you can invoke the Databricks REST API from Python, this might help you achieve what you’re aiming for.
I hope this helps!
--------------------------
Takuya Omi (尾美拓哉)
Takuya Omi (尾美拓哉)