Takuya-Omi
Valued Contributor III

Hello,

There are two ways to determine whether a DLT pipeline is running in Full Refresh or Incremental mode:

  1. DLT Event Log Schema
    The details column in the DLT event log schema includes information on "full_refresh". You can use this to identify whether it is True or False.

    DLT Event Log Schema Documentation

    An example of the details column is as follows:

    {"user_action":{"action":"START","user_name":"xxxxxxx@gmail.com","user_id":xxxxxxxx,"request":{"start_request":{"full_refresh":false,"validate_only":false}}}}
  2. Databricks REST API
    You can retrieve DLT pipeline information using the Databricks REST API, which also contains the "full_refresh" field. Here, you can check whether it is True or False.

    Since you can invoke the Databricks REST API from Python, this might help you achieve what you’re aiming for.

    Databricks REST API Documentation - Get Pipeline Update

I hope this helps!

--------------------------
Takuya Omi (尾美拓哉)