To trigger databricks workflow on defined frequency
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-25-2024 09:33 AM
Hi Databricks, I am trying to run a databricks workflow on scheduled basis (for e.g. the frequency is after every five mins). Here is the databricks.yaml file:
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-25-2024 10:00 AM
Hi @argl1995dbks ,
You are deploying in development mode. In this mode all schedules and trigger are paused.
If you would like unpause schedules and triggers you need to set schedule.pause_set to UNPAUSED.
Refer to documentation
Databricks Asset Bundle deployment modes | Databricks on AWS
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-25-2024 10:28 AM
Hi @szymon_dybczak , thanks for the help, the issue is sorted, however I have a question i.e. we have many workflows which we have to trigger on Monthly and Quarterly frequency, so is there a way where we can have a yaml file specifically for capturing the frequency and then reference it in our databricks.yaml, if YES can you please help me with that as well.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-25-2024 10:43 AM
Have a look on example section from documentation. The last example is showing how to modularize yaml files. I think you can try define schedule in separate yaml files and the reuse it by including.
https://learn.microsoft.com/en-us/azure/databricks/dev-tools/bundles/settings#--examples
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-25-2024 10:59 AM
Hi @szymon_dybczak , I have followed this documentation link today and created a new custom variables yaml file and refer it in the databricks.yaml and it got failed at the deployment phase. There is already a discussion going on for the same https://community.databricks.com/t5/machine-learning/passing-parameters-in-databricks-workflows/td-p...
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-26-2024 01:34 AM
Hi, let me explain you the current scenario, we have databricks workflows which has DS, DE and MLOps tasks. The workflows are meant to be triggered on a specific frequency i.e. Monthly and Quarterly and Quarterly workflow depends on the Monthly workflow. We have maintained a configuration file for it. Here is the example:
![](/skins/images/97567C72181EBE789E1F0FD869E4C89B/responsive_peak/images/icon_anonymous_message.png)
![](/skins/images/97567C72181EBE789E1F0FD869E4C89B/responsive_peak/images/icon_anonymous_message.png)