topic Re: How to Pass Dynamic Parameters (e.g., Current Date) in Databricks Workflow UI? in Get Started Discussions

How to Pass Dynamic Parameters (e.g., Current Date) in Databricks Workflow UI?

abubakar-saddiq — Thu, 17 Oct 2024 14:33:31 GMT

I'm setting up a job in the Databricks Workflow UI and I want to pass a dynamic parameter, like the current date (run_date), each time the job runs.

In Azure Data Factory, I can use expressions like @utcnow() to calculate this at runtime. However, I want to achieve the same in Databricks, where the run_date parameter would be automatically calculated and passed at runtime without having to hardcode or manually update it.

For example, in the notebook, I am currently using this to get the parameter:

run_date = dbutils.widgets.get("run_date")

Is there a way to configure dynamic parameters in the Databricks Workflow UI so that run_date gets calculated at runtime, or does this require external handling (e.g., through scripts or APIs)?

Re: How to Pass Dynamic Parameters (e.g., Current Date) in Databricks Workflow UI?

szymon_dybczak — Thu, 17 Oct 2024 15:34:14 GMT

Hi @abubakar-saddiq ,

Yes, you can use dynamic value reference in datbricks workflows:

https://learn.microsoft.com/en-us/azure/databricks/jobs/dynamic-value-references

Re: How to Pass Dynamic Parameters (e.g., Current Date) in Databricks Workflow UI?

-werners- — Fri, 18 Oct 2024 07:16:47 GMT

As szymon mentioned, dynamic parameter values exist, but the functionality is still far from what Data Factory has to offer.
I am pretty sure though that this will be extended.
So for the moment I suggest you do the value derivation in data factory, and pass the value to the databricks job (create an ADF pipeline to launch dbrx jobs using the API).