Can we parameterize the compute in job cluster

NarenderKumar
New Contributor III

‎07-01-2024 03:32 AM

I have created a workflow job in databricks with job parameters.

I want to run the job same with different workloads and data volume.

So I want the compute cluster to be parametrized so that I can pass the compute requirements(driver, executor size and number of nodes) dynamically when I run the job.

Is this possible in databricks?

Labels:
0 Kudos

raphaelblg
Databricks Employee
Databricks Employee

‎07-01-2024 10:26 AM

Hi @NarenderKumar , If you want to change an existing job compute you would have to update the job settings before triggering a new run. Feel free to open a feature request with your idea through the Databricks Ideas Portal.

 

Best regards,

Raphael Balogo
Sr. Technical Solutions Engineer
Databricks

View solution in original post

brockb
Databricks Employee
Databricks Employee

‎07-01-2024 10:30 AM

Hi @NarenderKumar ,

Have you considered leveraging autoscaling for the existing cluster?

If this does not meet your needs, are the differing volume/workloads known in advance? If so, could different compute be provisioned using Infrastructure as Code based on the differing workload characteristics? Here's a doc on using Terraform with Databricks: https://docs.databricks.com/en/dev-tools/terraform/index.html

Thank you.