topic Re: Apply expectations only if column exists in Data Engineering

Apply expectations only if column exists

Hoviedo — Thu, 05 Dec 2024 13:46:57 GMT

Hi, is there any way to apply a expectations only if that column exists? I am creating multiple dlt tables with the same python function so i would like to create diferent expectations based in the table name, currently i only can create expectations for those column that exists in all the tables created for the python function

thanks

Re: Apply expectations only if column exists

Walter_C — Thu, 05 Dec 2024 14:02:23 GMT

To apply expectations only if a column exists in Delta Live Tables (DLT), you can use the @Dlt.expect decorator conditionally within your Python function. Here is a step-by-step approach to achieve this:

Check if the Column Exists: Before applying the expectation, check if the column exists in the DataFrame.
Apply Expectations Conditionally: Use the @Dlt.expect decorator only if the column is present.

Re: Apply expectations only if column exists

Hoviedo — Thu, 05 Dec 2024 14:32:00 GMT

Hi Walter, sorry what would it look like in the code?

i can do this in the python function (it does nothing)

if "Distance" in df.columns:

dlt.expect("Distance is positive", "Distance >= 0")

but i am now sure how can i apply the same with the decorator

Re: Apply expectations only if column exists

Hoviedo — Thu, 05 Dec 2024 14:50:08 GMT

def get_changes_from_raw(table_name):

@Dlt.table(

name=f"{table_name}_changes",

comment=f"New {table_name} data incrementally ingested from cloud object storage landing zone",

)

@Dlt.expect("valid_rescued_data", "_rescued_data is null")

def read_changes_from_adl2():

df = spark.read...

if "distance" in df.columns:

dlt.expect("distance is positive", "distance >= 0")

Re: Apply expectations only if column exists

Fevemania — Fri, 11 Jul 2025 16:00:31 GMT

Hi @Walter_C ,

I wonder that how does the step-by-step approach work.

As far as I know, Decorators (@dlt.expect) are processed when the function code is first read and the pipeline's structure is being defined—this is the parsing phase. But what check if the column exists, looks like is during when Spark is actually processing data—the execution phase. As you can see at below, if column doesn't exist, the lakeflow declarative pipeline failed at initialization phase.

The method proposed by @Hoviedo might not work well, as I try to do the same thing, I found the metrics in UI is disappear.

Before (use the decorator):

After using non-decorator manner: