How to Pass Dynamic Parameters (e.g., Current Date) in Databricks Workflow UI?

[abubakar-saddiq]

I'm setting up a job in the Databricks Workflow UI and I want to pass a dynamic parameter, like the current date (run_date), each time the job runs.

In Azure Data Factory, I can use expressions like @utcnow() to calculate this at runtime. However, I want to achieve the same in Databricks, where the run_date parameter would be automatically calculated and passed at runtime without having to hardcode or manually update it.

For example, in the notebook, I am currently using this to get the parameter:

run_date = dbutils.widgets.get("run_date")

Is there a way to configure dynamic parameters in the Databricks Workflow UI so that run_date gets calculated at runtime, or does this require external handling (e.g., through scripts or APIs)?

[szymon_dybczak]

Hi @abubakar-saddiq ,

Yes, you can use dynamic value reference in datbricks workflows: 

https://learn.microsoft.com/en-us/azure/databricks/jobs/dynamic-value-references

[-werners-]

As szymon mentioned, dynamic parameter values exist, but the functionality is still far from what Data Factory has to offer.
I am pretty sure though that this will be extended.
So for the moment I suggest you do the value derivation in data factory, and pass the value to the databricks job (create an ADF pipeline to launch dbrx jobs using the API).