I would go to datadog as it is mentioned most time and has excellent integration https://docs.datadoghq.com/integrations/databricks/?tab=driveronly

, but there is one question that bothers me. As it is through init scripts, how to monitor shared Unity catalog clusters that don't have init scripts?

Thanks Dudek, Much appreciated on the response so quickly. Yes we were trying datadog, but unfortunately ours is on Azure platform and below is the reply we got from datadog support team. We are still following with them.

I'd recommend this new tool we've been trying out. It's really helpful for monitoring and provides good insights on how Azure Databricks clusters, pools & jobs are doing – like if they're healthy or having issues. It brings everything together, making it easier to figure out what's going wrong and fix it faster.

Personally I also feel they seem focused on Databricks but ultimately whatever tool provides your team the best visibility and automation should take priority. Check this video, maybe this might also help - https://www.youtube.com/watch?v=vD0S3h7ZJNU&t=355s