Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-13-2023 07:58 AM
@Dinu Sukumara : My take on your question -
If you want to preserve the timestamp values exactly as they are in the source database, without any timezone conversion, you can follow these steps:
- Set the Cluster Timezone: Keep the cluster timezone as UTC, as you mentioned.
- Adjust Session Timezone: Before reading the data from the source database, you can adjust the session timezone in Databricks to match the timezone of the source data.
spark.conf.set("spark.sql.session.timeZone", "<source_timezone>")3 . Read Data from Source: Use the JDBC read functionality in Databricks to extract the data from the source database.
df = spark.read.format("jdbc").option("url", "<jdbc_url>").option("dbtable", "<table_name>").load()