Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-26-2024 01:34 AM
Hi, let me explain you the current scenario, we have databricks workflows which has DS, DE and MLOps tasks. The workflows are meant to be triggered on a specific frequency i.e. Monthly and Quarterly and Quarterly workflow depends on the Monthly workflow. We have maintained a configuration file for it. Here is the example:
countries:
INDIA:
enabled_components: ds_de
calendar:
- [1, 2]
- [2, 4]
Africa:
enabled_components: all
calendar:
- [3, 4]
- [5, 6]
NA:
enabled_components: all
calendar:
- [4, 5]
- [4, 8]
Currently we are using databricks scheduler and will soon be planning to switch to Automic for triggering these workflows but I have a couple of questions i.e.
1. How to reference this config.yaml file in the databricks.yaml file I provided above?
2. How DAB can help improving the current set up?
