Apply expectations only if column exists

[Hoviedo]

Hi, is there any way to apply a expectations only if that column exists? I am creating multiple dlt tables with the same python function so i would like to create diferent expectations based in the table name, currently i only can create expectations for those column that exists in all the tables created for the python function

thanks

[Walter_C]

To apply expectations only if a column exists in Delta Live Tables (DLT), you can use the @Dlt.expect decorator conditionally within your Python function. Here is a step-by-step approach to achieve this:

1. Check if the Column Exists: Before applying the expectation, check if the column exists in the DataFrame.
2. Apply Expectations Conditionally: Use the @Dlt.expect decorator only if the column is present.

[Hoviedo]

Hi Walter, sorry what would it look like in the code?

i can do this in the python function (it does nothing)

if "Distance" in df.columns:

            dlt.expect("Distance is positive", "Distance >= 0")

but i am now sure how can i apply the same with the decorator

[Hoviedo]

def get_changes_from_raw(table_name):

    @Dlt.table(

        name=f"{table_name}_changes",

        comment=f"New {table_name} data incrementally ingested from cloud object storage landing zone",

    )

    @Dlt.expect("valid_rescued_data", "_rescued_data is null")

    def read_changes_from_adl2():

          df = spark.read...

          if "distance" in df.columns:

              dlt.expect("distance is positive", "distance >= 0")