cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Databricks Workflow Orchestration

shan-databricks
New Contributor II

I have 50 tables and will increase gradually, so I want to create a single workflow to orchestrate the job and run it table-wise. Is there an option to do this in Databricks workflow?

2 REPLIES 2

Edthehead
Contributor III

Breakup these 50 tables logically or functionally and place them in their own workflows. A good strategy would be to group tables that are dependent in the same workflow. Then use a master workflow to trigger each child workflow. So it will be like a branching workflow where a parent can trigger other workflow. A single workflow can have up to 100 tasks. But you can work around this by having 1 workflow trigger other workflows and in this way bypass that limit.

Thank you, I will try this.