How to update num_workers dynamically in a job cluster

[NickLee]

I am setting up a workflows with the UI. In the first task, a dynamic value for the next task's num_workers is calculated based on actual data size. In the subsequent task, I'd like to use this calculated num_workers to update the job cluster's default num_workers. Is this possible? Thanks.

[NickLee]

wonder if anyone has similar experience? thanks

[Kaniz_Fatma · Accepted solution]

Hi @NickLee, To dynamically update the num_workers in a job cluster within Azure Databricks, you can utilize the Jobs API 2.0. This allows you to modify job settings, including the number of worker nodes, based on dynamically calculated values from previous tasks in your workflow.

[NickLee]

Thank you! Will give it a try.