topic Re: one of the date datatype format issue in pysaprk in Data Engineering

one of the date datatype format issue in pysaprk

KVNARK — Wed, 11 Jan 2023 13:11:16 GMT

if anyone has encountered this date type format - 6/15/25 12:00 AM could you mention the right formatting to be used in Pyspark.

Thanks in advance!

UmaMahesh1 — Wed, 11 Jan 2023 13:49:58 GMT

Hi @KVNARK .

You can use the below formatting to get it into a timestamp format...

to_timestamp('6/15/25 12:00 AM', 'MM/dd/yy hh:mm aa')

This will give you the timestamp like this -- 2025-06-15T00:00:00.000+0000

If you want only the date part, you can use to_date function.

Hope this helps...Cheers..

UmaMahesh1 — Wed, 11 Jan 2023 13:58:01 GMT

FYI... You also have to set set spark.sql.legacy.timeParserPolicy = LEGACY for the above to work..

Hubert-Dudek — Wed, 11 Jan 2023 16:39:51 GMT

Without legacy, it will also work.

SELECT to_timestamp('6/15/23 12:00 AM', 'M/dd/yy h:mm a')