topic Re: Change {{job.start_time.[iso_date]}} Timezone in Data Engineering

Change {{job.start_time.[iso_date]}} Timezone

QuantumFries — Tue, 30 Apr 2024 02:09:23 GMT

I am trying to schedule some jobs using workflows and leveraging dynamic variables. One caveat is that when I try to use {{job.start_time.[iso_date]}} it seems to be defaulted to UTC, is there a way to change it?

Re: Change {{job.start_time.[iso_date]}} Timezone

Aviral-Bhardwaj — Tue, 30 Apr 2024 13:08:23 GMT

Try to use this configuration let me know your finding here

spark configuration page- https://spark.apache.org/docs/latest/configuration.html

Re: Change {{job.start_time.[iso_date]}} Timezone

artsheiko — Tue, 30 Apr 2024 14:19:26 GMT

Hi, all the dynamic values are in UTC (documentation).

Maybe you can use the code like the one presented below + pass the variables between tasks (see Share information between tasks in a Databricks job) ?

%python from datetime import datetime, timedelta # Get the current date current_date = datetime.now() # Subtract one day to get the previous day or make any other shift previous_day = current_date - timedelta(days=1) # Format the date as a string in the desired format if needed previous_day_str = previous_day.strftime('%Y-%m-%d') # Now you can use previous_day_str as a dynamic variable in your notebook logic

Re: Change {{job.start_time.[iso_date]}} Timezone

QuantumFries — Thu, 02 May 2024 14:33:38 GMT

The configurations did not work. Once the job cluster starts, it leverages the input which is already set. I did not try to set this in the init_script, but I believe it would be the same behavior.

Re: Change {{job.start_time.[iso_date]}} Timezone

QuantumFries — Thu, 02 May 2024 14:35:10 GMT

This does work, I think I will leverage something like this even thought it is a bit hacky.