Hi I am struggling with converting a timestamp string with T and Z to a timestamp column in my DLT job.  here is the relevent code snippet:

trans_rules={'timestamp_value', '''to_timestamp(timstamp_str, "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'")'''}

In my DLT function, I have the following:

for col_name, expr_str in transformation_rules.items():

df = df.withColumn(col_name, F.expr(expr_str))

 

The DLT execution succeeded but the target column (timestamp_value) is null.

 

 

However, when I execute the similar command in a notebook, it works fine.

 

I am using serverless compute for DLT