How to get different dynamic value for each task in workflow

[mh_db]

I created a workflow with two tasks. It runs the first notebook and then it wait for that to finish to start the second notebook. I want to use this dynamic value as one of the parameters {{job.start_time.iso_datetime}} for both tasks. This should give me the date and time. I noticed when I run the job it will take the current date and time and use it for the first task/notebook but not sure why it takes the same date and time as the first task to run the second task/notebook as well? Shouldn't it use the date and time the second task started to run?

I need the time for first and second task to be different . Is there a way to do this in the workflow?

[lucasrocha]

Hello @mh_db ,

The dynamic value {{job.start_time.iso_datetime}} you are using in your workflow is designed to capture the start time of the job run, not the individual tasks within the job. This is why you are seeing the same date and time for both tasks, as they are part of the same job run.

If you want to capture the start time of each individual task, currently Databricks does not have this dynamic value on the supported value reference. However, you can use a workaround by capturing the current date and time at the beginning of each task in the notebook code itself. This will give you the start time of each task when it begins to run.

For example, in Python, you can use the following code to get the current date and time:

—————————————————
from datetime import datetime

start_time = datetime.now()

print("Start time: ", start_time)
—————————————————

You can then use this start_time variable throughout your notebook as needed. Please note that this time will be in the timezone of the Databricks cluster running the job.

You can find the supported value references and its description through the doc below.
https://docs.databricks.com/en/workflows/jobs/parameter-value-references.html#supported-value-refere...

If you believe that having a dynamic value reference for the task run start time would be beneficial, you could submit an idea in Databricks' Ideas Portal.

Best regards,
Lucas Rocha
Technical Solutions Engineer
Databricks