cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Azure Databricks Monitoring & Alerting (Data Observability) Tools / Frameworks for Enterprise

pSdatabricks
New Contributor II

I am trying to evaluate options for Monitoring and Alerting tools like New Relic, Datadog, Grafana with Databricks on Azure . No one supports when reached out to them. I would like to hear from the databricks team on the recommended tool / framework ?

3 REPLIES 3

Hubert-Dudek
Esteemed Contributor III

I would go to datadog as it is mentioned most time and has excellent integration https://docs.datadoghq.com/integrations/databricks/?tab=driveronly

, but there is one question that bothers me. As it is through init scripts, how to monitor shared Unity catalog clusters that don't have init scripts?

pSdatabricks
New Contributor II

Thanks Dudek, Much appreciated on the response so quickly. Yes we were trying datadog, but unfortunately ours is on Azure platform and below is the reply we got from datadog support team. We are still following with them.

image.png

Sruthivika
New Contributor II

I'd recommend this new tool we've been trying out. It's really helpful for monitoring and provides good insights on how Azure Databricks clusters, pools & jobs are doing – like if they're healthy or having issues. It brings everything together, making it easier to figure out what's going wrong and fix it faster.

Personally I also feel they seem focused on Databricks but ultimately whatever tool provides your team the best visibility and automation should take priority. Check this video, maybe this might also help - https://www.youtube.com/watch?v=vD0S3h7ZJNU&t=355s

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group