We need to pull data from an api endpoint at every 1 minute interval. Time to fetch and process is 30 seconds. 
if we use job cluster once the process completes it get terminated and for next run we have to wait ~3-5 mins to spin up the cluster. Which over pass our SLA 1 minute.