Databricks Workflow Orchestration

shan-databricks
Databricks Partner

I have 50 tables and will increase gradually, so I want to create a single workflow to orchestrate the job and run it table-wise. Is there an option to do this in Databricks workflow?

Edthehead
Contributor III

Breakup these 50 tables logically or functionally and place them in their own workflows. A good strategy would be to group tables that are dependent in the same workflow. Then use a master workflow to trigger each child workflow. So it will be like a branching workflow where a parent can trigger other workflow. A single workflow can have up to 100 tasks. But you can work around this by having 1 workflow trigger other workflows and in this way bypass that limit.

Thank you, I will try this.