topic Re: Possibility of creating and running concurrent Job Runs from a single job all parameters driven in Data Engineering

Possibility of creating and running concurrent Job Runs from a single job all parameters driven

iskidet_glenny — Mon, 14 Apr 2025 01:58:58 GMT

Hello Community,

I hope everyone is doing well.

I’ve been exploring the idea of creating multiple instances of a job which will be jobs runs with different parameter configurations. Has anyone else considered this approach?

Imagine a scenario where you need to backfill data to a data lake. Instead of fanning out with multiple tasks, what if we could create multiple independent instances of a job run? These instances would be completely isolated from each other, yet target the same base storage, each driven by unique parameters—similar to the principles of standard object-oriented programming (OOP).

Re: Possibility of creating and running concurrent Job Runs from a single job all parameters driven

SP_6721 — Mon, 14 Apr 2025 17:27:16 GMT

Hi @iskidet_glenny

Yes, running multiple instances of a Databricks job with different parameters is a common and solid approach especially when it comes to backfilling data.

So usually, we set up one job and just pass in different parameters each time we run it. No need to create a bunch of separate jobs. Then we trigger the runs using the Databricks API. That way, we can start many runs at once, each with its own settings.

All the jobs run in parallel, on their own. If they’re writing to the same place, just make sure they don’t mess each other up or overwrite anything.

A few things to watch out for your cluster should be able to handle the load if we are running a lot of jobs at the same time. And it’s always a good idea to check each job’s logs and status to catch any issues.

Re: Possibility of creating and running concurrent Job Runs from a single job all parameters driven

Roshaan — Thu, 17 Jul 2025 00:07:14 GMT

I have seen correlation that bigger the cluster configuration leads to more concurrent job runs successfully, is that true and if so why?