Hi I am struggling with converting a timestamp string with T and Z to a timestamp column in my DLT job. here is the relevent code snippet:
trans_rules={'timestamp_value', '''to_timestamp(timstamp_str, "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'")'''}
In my DLT function, I have the following:
for col_name, expr_str in transformation_rules.items():
df = df.withColumn(col_name, F.expr(expr_str))
The DLT execution succeeded but the target column (timestamp_value) is null.
However, when I execute the similar command in a notebook, it works fine.
I am using serverless compute for DLT
