spark 3.4 and databricks 13 introduce two new types of timestamps for handling time zone information:- TIMESTAMP WITH LOCAL TIME ZONE: This type assum...

[Hubert-Dudek]

spark 3.4 and databricks 13 introduce two new types of timestamps for handling time zone information:

- TIMESTAMP WITH LOCAL TIME ZONE: This type assumes that the input data is in the session's local time zone and converts it to UTC before processing. This allows for consistent results across different time zones and daylight saving changes.

- TIMESTAMP WITHOUT TIME ZONE: This type treats the input data as time zone insensitive and performs no conversion. This is useful for data that does not depend on the time zones context, such as event logs or sensor readings.

 

[Anonymous]

This is helpful! Timestamps are always the reason to mess up the business logic as we know.