topic Re: Databricks Workflow Orchestration in Data Engineering

Databricks Workflow Orchestration

shan-databricks — Thu, 13 Feb 2025 12:18:59 GMT

I have 50 tables and will increase gradually, so I want to create a single workflow to orchestrate the job and run it table-wise. Is there an option to do this in Databricks workflow?

Re: Databricks Workflow Orchestration

Edthehead — Fri, 14 Feb 2025 08:06:08 GMT

Breakup these 50 tables logically or functionally and place them in their own workflows. A good strategy would be to group tables that are dependent in the same workflow. Then use a master workflow to trigger each child workflow. So it will be like a branching workflow where a parent can trigger other workflow. A single workflow can have up to 100 tasks. But you can work around this by having 1 workflow trigger other workflows and in this way bypass that limit.

Re: Databricks Workflow Orchestration

shan-databricks — Mon, 17 Feb 2025 02:21:03 GMT

Thank you, I will try this.