one of the date datatype format issue in pysaprk
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-11-2023 05:11 AM
if anyone has encountered this date type format - 6/15/25 12:00 AM could you mention the right formatting to be used in Pyspark.
Thanks in advance!
- Labels:
-
Date
-
Format Issue
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-11-2023 05:49 AM
Hi @KVNARK .
You can use the below formatting to get it into a timestamp format...
to_timestamp('6/15/25 12:00 AM', 'MM/dd/yy hh:mm aa')
This will give you the timestamp like this -- 2025-06-15T00:00:00.000+0000
If you want only the date part, you can use to_date function.
Hope this helps...Cheers..
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-11-2023 05:58 AM
FYI... You also have to set set spark.sql.legacy.timeParserPolicy = LEGACY for the above to work..
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-11-2023 08:39 AM
Without legacy, it will also work.
SELECT to_timestamp('6/15/23 12:00 AM', 'M/dd/yy h:mm a')