Databricks Community

Comment

New Contributor II

Apologies my above comment on the datetime format is wrong of course, the format in the article is correct, I just forgot to add the legacy timeParserPolicy settings.

#To Parse Timestamps correctly

spark.conf.set("spark.sql.legacy.timeParserPolicy", "LEGACY")

All datetime formats parsed:

df = df.withColumn("SinkCreatedOn", to_timestamp(col("SinkCreatedOn"), "MM/dd/yyyy hh:mm:ss aa")

df = df.withColumn("SinkModifiedOn", to_timestamp(col("SinkModifiedOn"), "MM/dd/yyyy hh:mm:ss aa"))

df = df.withColumn("modifieddatetime", to_timestamp(col("modifiedon"), "yyyy-MM-dd'T'HH:mm:ss.SSSSSSS'Z'"))

df = df.withColumn("createddatetime", to_timestamp(col("modifiedon"), "yyyy-MM-dd'T'HH:mm:ss.SSSSSSS'Z'"))

df = df.withColumn("createdon", to_timestamp(col("createdon"), "yyyy-MM-dd'T'HH:mm:ss.SSSSSSSXXX"))

df = df.withColumn("modifiedon", to_timestamp(col("modifiedon"), "yyyy-MM-dd'T'HH:mm:ss.SSSSSSS'Z'"))