topic What is the maximum of concurrent streaming jobs for a cluster? in Data Engineering

What is the maximum of concurrent streaming jobs for a cluster?

jwilliam — Wed, 31 Aug 2022 04:25:48 GMT

What is the maximum of concurrent streaming jobs for a cluster? How can I have the right amount of concurrent streaming jobs for different cluster configuration?

Should I use multiple cluster for different jobs or combine it into a big cluster to handle all the jobs?

Re: What is the maximum of concurrent streaming jobs for a cluster?

Prabakar — Thu, 01 Sep 2022 12:22:05 GMT

Hi @John William it would be better to use different clusters for each streaming jobs.

Re: What is the maximum of concurrent streaming jobs for a cluster?

jwilliam — Mon, 05 Sep 2022 04:20:24 GMT

I worried about the cost of this approach, spin up new cluster for every streaming job running non stop required a lot of resources.

Re: What is the maximum of concurrent streaming jobs for a cluster?

Prabakar — Mon, 05 Sep 2022 08:39:48 GMT

I understand, but you can calculate the risk involved in using a single cluster for all your streaming jobs. Let's say you are running 4 streaming jobs in a cluster and because of 1 job the cluster gets into a hung state or something went wrong on the cluster, then all 4 jobs will be affected. However, if you use separate clusters for each streaming job, then in the event of such problems only one job will be affected and others will be running properly. This is my thought. You need to decide all factors and plan the clusters. Also you can calculate the pricing for one cluster and multiple clusters.

Let's say for 4 streaming jobs, I use a single cluster of i3.4xlarge instance with 10 workers of the same type, I use 44 DBU/hr

& if I use 1 cluster per job, so I can use 4 smaller clusters each i3.xlarge instance with 10 workers will also cost me 44 DBU/hour (11 DBU/hr per cluster).

This way you can calculate the workload and the pricing and decide on the cluster sizing.