Automatic conversion of timestamp to the default timezone
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-18-2024 07:48 AM
I am encountering the issue when ingesting data from adls xml or json files to process them via Pyspark (Autoloader or just reading df). The timestamp is automatically converted to the default timezone.And I have dynamically timezone values. Did anyone of you has found a way how to not let the conversion happened ? I tried to set the
spark.conf.set("spark.sql.session.timeZone", "UTC"), but it is not working.
Original local time: 2024-06-21 20:50:00
the Offset: +08:00 : This means the local time is 8 hours ahead of UTC.
Subtract 8 hours: 2024-06-21 20:50:00 - 8:00:00 = 2024-06-21 12:50:00
Result in UTC: 2024-06-21T12:50:00Z
