Databricks Asset Bundle conditional job cluster size?

[Bart_DE]

Hey folks,

Can someone please suggest if there is a way to spawn a job cluster of a given size if a parameter of the job invocation (e.g file_name) contains a desired value? I have a job which 90% of the time deals with very small files, but the remaining 10% requires much bigger machines. How can i define job.yml file in such a way? Is this even possible to be done without some pre-process that checks the file name?

Thanks!

[lingareddy_Alva · Accepted solution]

Hi @Bart_DE 

No — a single job.yml file can’t “look inside” a parameter like file_name and then decide to spin up a different job-cluster size on the fly.
Job-cluster definitions in Databricks Workflows (Jobs) are static. All the heavy-lifting has to be done outside the cluster specification: