DLT job failed to parse timestamp string with T and Z.

[Thomas_Zhang]

Hi I am struggling with converting a timestamp string with T and Z to a timestamp column in my DLT job.  here is the relevent code snippet:

trans_rules={'timestamp_value', '''to_timestamp(timstamp_str, "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'")'''}

In my DLT function, I have the following:

for col_name, expr_str in transformation_rules.items():

df = df.withColumn(col_name, F.expr(expr_str))

The DLT execution succeeded but the target column (timestamp_value) is null.

However, when I execute the similar command in a notebook, it works fine.

 

I am using serverless compute for DLT

[Ayushi_Suthar]

Hi @Thomas_Zhang , Good Day!

Can you give it a try with the below code format and check if it helps you to set the timezone with Z : 

=======================

date_format( to_timestamp(`createdon`, 'yyyy-MM-dd\'T\'HH:mm:ss.SSSSSSSX'), 'yyyy-MM-dd\'T\'HH:mm:ss.SSSSSSSX' )

=======================

Please let me know if this helps and leave a like if this information is useful, followups are appreciated.
Kudos
Ayushi