How to update num_workers dynamically in a job cluster

NickLee
New Contributor III

I am setting up a workflows with the UI. In the first task, a dynamic value for the next task's num_workers is calculated based on actual data size. In the subsequent task, I'd like to use this calculated num_workers to update the job cluster's default num_workers. Is this possible? Thanks.

NickLee_0-1722018584496.png

 

NickLee
New Contributor III

wonder if anyone has similar experience? thanks

NickLee
New Contributor III

Thank you! Will give it a try.