one of the date datatype format issue in pysaprk

KVNARK
Honored Contributor II

if anyone has encountered this date type format - 6/15/25 12:00 AM could you mention the right formatting to be used in Pyspark.

Thanks in advance!

UmaMahesh1
Honored Contributor III

Hi @KVNARK .​ 

You can use the below formatting to get it into a timestamp format...

to_timestamp('6/15/25 12:00 AM', 'MM/dd/yy hh:mm aa')

This will give you the timestamp like this -- 2025-06-15T00:00:00.000+0000

If you want only the date part, you can use to_date function.

Hope this helps...Cheers..

Uma Mahesh D

UmaMahesh1
Honored Contributor III

FYI... You also have to set set spark.sql.legacy.timeParserPolicy = LEGACY for the above to work..

Uma Mahesh D

Hubert-Dudek
Databricks MVP

Without legacy, it will also work.image.png

SELECT to_timestamp('6/15/23 12:00 AM', 'M/dd/yy h:mm a')


My blog: https://databrickster.medium.com/