Apply expectations only if column exists

[Hoviedo]

Hi, is there any way to apply a expectations only if that column exists? I am creating multiple dlt tables with the same python function so i would like to create diferent expectations based in the table name, currently i only can create expectations for those column that exists in all the tables created for the python function

thanks

[Walter_C]

To apply expectations only if a column exists in Delta Live Tables (DLT), you can use the @Dlt.expect decorator conditionally within your Python function. Here is a step-by-step approach to achieve this:

1. Check if the Column Exists: Before applying the expectation, check if the column exists in the DataFrame.
2. Apply Expectations Conditionally: Use the @Dlt.expect decorator only if the column is present.

[Hoviedo]

Hi Walter, sorry what would it look like in the code?

i can do this in the python function (it does nothing)

if "Distance" in df.columns:

            dlt.expect("Distance is positive", "Distance >= 0")

but i am now sure how can i apply the same with the decorator

[Hoviedo]

def get_changes_from_raw(table_name):

    @Dlt.table(

        name=f"{table_name}_changes",

        comment=f"New {table_name} data incrementally ingested from cloud object storage landing zone",

    )

    @Dlt.expect("valid_rescued_data", "_rescued_data is null")

    def read_changes_from_adl2():

          df = spark.read...

          if "distance" in df.columns:

              dlt.expect("distance is positive", "distance >= 0")

[Fevemania]

Hi @Walter_C ,

I wonder that how does the step-by-step approach work. 

As far as I know, Decorators (@dlt.expect) are processed when the function code is first read and the pipeline's structure is being defined—this is the parsing phase. But what check if the column exists, looks like is during when Spark is actually processing data—the execution phase. As you can see at below, if column doesn't exist, the lakeflow declarative pipeline failed at initialization phase.

The method proposed by @Hoviedo might not work well, as I try to do the same thing, I found the metrics in UI is disappear.
 
Before (use the decorator): 

 

After using non-decorator manner: