โ06-08-2023 12:43 AM
Timestamp columns which are extracted from source databases using jdbc read are getting converted to different timezone and is not matching with source timestamp. Could anyone suggest how can we get same timestamp data like source data?
โ06-08-2023 01:10 AM
can you check this?
โ06-08-2023 11:28 AM
Thank You, if I am keeping cluster timezone as "UTC" it is converting all the timestamp to UTC. What I am looking is to get timestamp field value in source same in dataframe. Could you please let me know if you have any suggestions?
โ06-13-2023 07:57 AM
@Werner Stinckensโ : Would you like to take this ahead? ๐
โ06-13-2023 07:58 AM
@Dinu Sukumaraโ : My take on your question -
If you want to preserve the timestamp values exactly as they are in the source database, without any timezone conversion, you can follow these steps:
spark.conf.set("spark.sql.session.timeZone", "<source_timezone>")
3 . Read Data from Source: Use the JDBC read functionality in Databricks to extract the data from the source database.
df = spark.read.format("jdbc").option("url", "<jdbc_url>").option("dbtable", "<table_name>").load()
โ06-13-2023 08:02 AM
Besides the options mentioned earlier there is also the convert_timezone function.
โ06-13-2023 11:34 PM
Hi @Dinu Sukumaraโ
We haven't heard from you since the last response from @Werner Stinckensโ โ . Kindly share the information with us, and in return, we will provide you with the necessary solution.
Thanks and Regards
โ06-14-2023 01:27 AM
Hi @Vidula Khannaโ @Suteja Kanuriโ @Werner Stinckensโ after changing the tz on cluster level to UTC I am getting the source data as expected, but where I am using current timestamp in databricks I need to use from_utc_timestamp function. that requires more changes. So I am still checking if there are any other workarounds? Also can we give TZ o databricks database level or so? Any suggestions please?
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโt want to miss the chance to attend and share knowledge.
If there isnโt a group near you, start one and help create a community that brings people together.
Request a New Group