parallel run in job pipeline

Geeya — Wed, 14 Dec 2022 01:30:13 GMT

I am trying to build a pipeline which deploys a ML model, and I want to build the pipeline in Workflow/jobs.

In task of prediction of the model, I have hundreds of groups of input features, I use a for loop to get one group of input features and do prediction each time. Those groups are independent and the sequence of running doesn't matter. I want to set a threshold like 10, and kick off several parallel runs, each run will do prediction of 10 groups of input features. (If there are 100 groups, then 10 parallel runs; if there are 175 groups, then 18 runs).

Is there any method to make one take of a pipeline kick off several runs with different parameters and the number of runs is decided by input data size?

topic parallel run in job pipeline in Machine Learning

parallel run in job pipeline