Thanks @Walter_C it worked on changing the configuration as you have suggested.
One follow up question -  what does "Shared" access mode mean for a job cluster, is there a way to share a job cluster with other workloads? if so, it would be really interesting and would be good to know and optimize our stack, could you please share any resources