cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Delta save timestamp as timestamp with time zone

BorislavBlagoev
Valued Contributor III

Hello! I have the following problem. I want to save the delta table and that table contains timestamp columns, but when I try to write that table with spark the timestamp columns become timestamp with the time zone. This is a problem in my case because when I try to read the table from Trino the columns are in timestamp with time zone format. Does anyone know how I can change this behavior?

1 ACCEPTED SOLUTION

Accepted Solutions

Hubert-Dudek
Esteemed Contributor III

in Spark sql "Spark SQL defines the timestamp type as

TIMESTAMP WITH SESSION TIME ZONE"

you can check your timezone settings. Sometimes in such a case can be more comfortable to store timestamp as Long.

Here is more info about timezones https://docs.databricks.com/spark/latest/dataframes-datasets/dates-timestamps.html

If you share example file and code I could help more.

View solution in original post

4 REPLIES 4

Hubert-Dudek
Esteemed Contributor III

in Spark sql "Spark SQL defines the timestamp type as

TIMESTAMP WITH SESSION TIME ZONE"

you can check your timezone settings. Sometimes in such a case can be more comfortable to store timestamp as Long.

Here is more info about timezones https://docs.databricks.com/spark/latest/dataframes-datasets/dates-timestamps.html

If you share example file and code I could help more.

@Hubert Dudek​ This is happening if I use Delta. If I use parquet spark save the column in timestamp without time zone.

Kaniz
Community Manager
Community Manager

Hi, @Borislav Blagoev​  , Just a friendly follow-up. Do you still need help, or @Hubert Dudek (Customer)​ 's response help you to find the solution? Please let us know.

Bhimaniheet
New Contributor II

Hi @Hubert Dudek​ ,

image.pngWhen you have time, I describe my problem. Can you please check?

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.