Atanu
Databricks Employee
Databricks Employee

for adf or job run we always prefer job cluster. but for streaming, you may consider using interactive cluster . but anyway you need to monitor the cluster load, if loads are high there will be chance to job slowness as well as failure. also data size will be a factor. @nafri A​ 

View solution in original post