topic Re: Azure Databricks Monitoring & Alerting (Data Observability) Tools / Frameworks for Enterprise in Data Engineering

Azure Databricks Monitoring & Alerting (Data Observability) Tools / Frameworks for Enterprise

pSdatabricks — Mon, 30 Jan 2023 08:28:09 GMT

I am trying to evaluate options for Monitoring and Alerting tools like New Relic, Datadog, Grafana with Databricks on Azure . No one supports when reached out to them. I would like to hear from the databricks team on the recommended tool / framework ?

Re: Azure Databricks Monitoring & Alerting (Data Observability) Tools / Frameworks for Enterprise

Hubert-Dudek — Mon, 30 Jan 2023 08:34:22 GMT

I would go to datadog as it is mentioned most time and has excellent integration https://docs.datadoghq.com/integrations/databricks/?tab=driveronly

, but there is one question that bothers me. As it is through init scripts, how to monitor shared Unity catalog clusters that don't have init scripts?

Re: Azure Databricks Monitoring & Alerting (Data Observability) Tools / Frameworks for Enterprise

pSdatabricks — Mon, 30 Jan 2023 08:37:11 GMT

Thanks Dudek, Much appreciated on the response so quickly. Yes we were trying datadog, but unfortunately ours is on Azure platform and below is the reply we got from datadog support team. We are still following with them.

Re: Azure Databricks Monitoring & Alerting (Data Observability) Tools / Frameworks for Enterprise

Sruthivika — Fri, 29 Mar 2024 12:21:21 GMT

I'd recommend this new tool we've been trying out. It's really helpful for monitoring and provides good insights on how Azure Databricks clusters, pools & jobs are doing – like if they're healthy or having issues. It brings everything together, making it easier to figure out what's going wrong and fix it faster.

Personally I also feel they seem focused on Databricks but ultimately whatever tool provides your team the best visibility and automation should take priority. Check this video, maybe this might also help - https://www.youtube.com/watch?v=vD0S3h7ZJNU&t=355s