Databricks Asset Bundle conditional job cluster size?

Bart_DE — Tue, 27 May 2025 08:35:09 GMT

Hey folks,

Can someone please suggest if there is a way to spawn a job cluster of a given size if a parameter of the job invocation (e.g file_name) contains a desired value? I have a job which 90% of the time deals with very small files, but the remaining 10% requires much bigger machines. How can i define job.yml file in such a way? Is this even possible to be done without some pre-process that checks the file name?

Thanks!

Re: Databricks Asset Bundle conditional job cluster size?

lingareddy_Alva — Tue, 27 May 2025 19:06:56 GMT

Hi @Bart_DE

No — a single job.yml file can’t “look inside” a parameter like file_name and then decide to spin up a different job-cluster size on the fly.
Job-cluster definitions in Databricks Workflows (Jobs) are static. All the heavy-lifting has to be done outside the cluster specification:

topic Databricks Asset Bundle conditional job cluster size? in Data Engineering

Databricks Asset Bundle conditional job cluster size?

Re: Databricks Asset Bundle conditional job cluster size?