from_utc_timestamp silently double-shifts time when session timezone isn't UTC — docs should call th

rikkyvai — Wed, 08 Jul 2026 05:52:08 GMT

Summary:
from_utc_timestamp(current_timestamp(), '<tz>') produces an incorrect (future-shifted) timestamp whenever the Spark session timezone is already set to something other than UTC. This is a very common pattern for teams stamping dp_load_ts/created_at/audit columns, and the current docs don't warn against it.

Root cause:
current_timestamp() returns the current time in the session's configured timezone (spark.sql.session.timeZone / cluster timezone setting), not UTC. from_utc_timestamp(ts, tz) always treats its input ts as if it were UTC, regardless of what it actually is, and converts it into tz.

So if a cluster's session timezone is set to Asia/Bangkok (UTC+7):

SELECT current_timezone(); -- Asia/Bangkok SELECT current_timestamp() AS raw_ts, -- correct Bangkok wall-clock time from_utc_timestamp(current_timestamp(), 'Asia/Bangkok') AS double_shifted -- Bangkok time + 7 more hours

double_shifted ends up 7 hours ahead of the actual current time, because current_timestamp() is already local, and from_utc_timestamp adds the offset a second time on top of it.

Why this matters:
from_utc_timestamp(current_timestamp(), '<local tz>') is a very natural-looking pattern to write when a team wants a "load timestamp in our local timezone," and it's silently wrong whenever the session timezone isn't UTC. It produces no error — just a plausible-looking but incorrect future timestamp — so it can go unnoticed in production pipelines for a long time.

Suggested fix:
Add an explicit warning/example on the from_utc_timestamp (and ideally current_timestamp()) doc pages, something like:

⚠️ from_utc_timestamp always assumes its input is UTC. If your session timezone (current_timezone()) is not UTC, do not pass current_timestamp() directly into from_utc_timestamp() — this will double-apply the timezone offset. If the session timezone already matches your target timezone, use current_timestamp() directly instead.

topic from_utc_timestamp silently double-shifts time when session timezone isn't UTC — docs should call th in Data Engineering

from_utc_timestamp silently double-shifts time when session timezone isn't UTC — docs should call th