Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-11-2023 11:35 PM
f.e. there is a spark option to enable the 'old' date handling.
You can set spark.sql.legacy.timeParserPolicy to LEGACY to restore the behavior before Spark 3.0.
Frankly I am not a fan of that approach as Spark 3 gives you a lot of interesting date functions.
So what you could do is to first identify where you have date columns, and explicitly cast them to dates with the to_date function.