Programmatically set minimum workers for a job cluster based on file size?

Alena
New Contributor II

‎08-05-2025 01:41 PM

I’m running an ingestion pipeline with a Databricks job:

  1. A file lands in S3

  2. A Lambda is triggered

  3. The Lambda runs a Databricks job

The incoming files vary a lot in size, which makes processing times vary as well. My job cluster has autoscaling enabled, but scaling up takes time.

Ideally, if a 10 GB file comes in, I’d like the job to start with more than one worker immediately, instead of waiting for autoscaling to kick in.

I’m currently using the run-now API to trigger the job, but I don’t see a way to adjust the job cluster configuration at runtime.

Is there a way to programmatically set the minimum number of workers for a job cluster depending on the incoming file size?

 

0 Kudos

kerem
Contributor

‎08-05-2025 04:27 PM

Hi Alena, 

Jobs API has update functionality to be able to do that: https://docs.databricks.com/api/workspace/jobs_21/update

If for some reason you can’t update your pipeline before you trigger it you can also consider creating a new job with desired configuration every time you run a trigger (POST /api/2.2/jobs/create). 

 

Kerem Durak

0 Kudos