cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Delta save timestamp as timestamp with time zone

BorislavBlagoev
Valued Contributor III

Hello! I have the following problem. I want to save the delta table and that table contains timestamp columns, but when I try to write that table with spark the timestamp columns become timestamp with the time zone. This is a problem in my case because when I try to read the table from Trino the columns are in timestamp with time zone format. Does anyone know how I can change this behavior?

1 ACCEPTED SOLUTION

Accepted Solutions

Hubert-Dudek
Esteemed Contributor III

in Spark sql "Spark SQL defines the timestamp type as

TIMESTAMP WITH SESSION TIME ZONE"

you can check your timezone settings. Sometimes in such a case can be more comfortable to store timestamp as Long.

Here is more info about timezones https://docs.databricks.com/spark/latest/dataframes-datasets/dates-timestamps.html

If you share example file and code I could help more.

View solution in original post

3 REPLIES 3

Hubert-Dudek
Esteemed Contributor III

in Spark sql "Spark SQL defines the timestamp type as

TIMESTAMP WITH SESSION TIME ZONE"

you can check your timezone settings. Sometimes in such a case can be more comfortable to store timestamp as Long.

Here is more info about timezones https://docs.databricks.com/spark/latest/dataframes-datasets/dates-timestamps.html

If you share example file and code I could help more.

@Hubert Dudekโ€‹ This is happening if I use Delta. If I use parquet spark save the column in timestamp without time zone.

Bhimaniheet
New Contributor II

Hi @Hubert Dudekโ€‹ ,

image.pngWhen you have time, I describe my problem. Can you please check?

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group