A Job "pool"? (or task pool)

spott_submittab — Fri, 04 Nov 2022 22:46:45 GMT

I'm trying to run a single job multiple times with different parameters where the number of concurrent jobs is less than the number of parameters.

I have a job (or task...) J that takes parameter set p, I have 100 p values I want to run, however I only want 10 to run at a time (say I have 10 clusters, or I want to run them all on the same cluster and that cluster needs to have enough compute to run them concurrently, etc.), however I want all 100 to eventually run.

Is this possible? The maximum concurrent runs parameter just skips extra jobs and tasks without dependencies all run at the same time.

I'm aware that there is frequently a way to do something like this is spark, but that feels heavy handed (and requires rearchitecting my code).

Re: A Job "pool"? (or task pool)

Aviral-Bhardwaj — Sun, 18 Dec 2022 07:37:38 GMT

this is something new,interesting question, try to reach out databricks support team, maybe they have some good idea here

topic Re: A Job "pool"? (or task pool) in Data Engineering

A Job "pool"? (or task pool)

Re: A Job "pool"? (or task pool)