How to Pass Dynamic Parameters (e.g., Current Date) in Databricks Workflow UI?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-17-2024 07:33 AM
I'm setting up a job in the Databricks Workflow UI and I want to pass a dynamic parameter, like the current date (run_date), each time the job runs.
In Azure Data Factory, I can use expressions like @utcnow() to calculate this at runtime. However, I want to achieve the same in Databricks, where the run_date parameter would be automatically calculated and passed at runtime without having to hardcode or manually update it.
For example, in the notebook, I am currently using this to get the parameter:
run_date = dbutils.widgets.get("run_date")
Is there a way to configure dynamic parameters in the Databricks Workflow UI so that run_date gets calculated at runtime, or does this require external handling (e.g., through scripts or APIs)?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-17-2024 08:34 AM
Hi @abubakar-saddiq ,
Yes, you can use dynamic value reference in datbricks workflows:
https://learn.microsoft.com/en-us/azure/databricks/jobs/dynamic-value-references
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-18-2024 12:16 AM
As szymon mentioned, dynamic parameter values exist, but the functionality is still far from what Data Factory has to offer.
I am pretty sure though that this will be extended.
So for the moment I suggest you do the value derivation in data factory, and pass the value to the databricks job (create an ADF pipeline to launch dbrx jobs using the API).

