topic Re: Delta save timestamp as timestamp with time zone in Data Engineering

Delta save timestamp as timestamp with time zone

BorislavBlagoev — Wed, 09 Mar 2022 12:43:47 GMT

Hello! I have the following problem. I want to save the delta table and that table contains timestamp columns, but when I try to write that table with spark the timestamp columns become timestamp with the time zone. This is a problem in my case because when I try to read the table from Trino the columns are in timestamp with time zone format. Does anyone know how I can change this behavior?

Re: Delta save timestamp as timestamp with time zone

Hubert-Dudek — Wed, 09 Mar 2022 13:00:58 GMT

in Spark sql "Spark SQL defines the timestamp type as

TIMESTAMP WITH SESSION TIME ZONE"

you can check your timezone settings. Sometimes in such a case can be more comfortable to store timestamp as Long.

Here is more info about timezones https://docs.databricks.com/spark/latest/dataframes-datasets/dates-timestamps.html

If you share example file and code I could help more.

Re: Delta save timestamp as timestamp with time zone

BorislavBlagoev — Wed, 09 Mar 2022 13:13:49 GMT

@Hubert Dudek This is happening if I use Delta. If I use parquet spark save the column in timestamp without time zone.

Re: Delta save timestamp as timestamp with time zone

Bhimaniheet — Thu, 22 Jun 2023 12:12:05 GMT

Hi @Hubert Dudek ,

When you have time, I describe my problem. Can you please check?