topic Re: Azure Databricks Status in Administration & Architecture

Azure Databricks Status

noorbasha534 — Fri, 25 Apr 2025 13:07:15 GMT

Dear all,

I wanted to check if anyone implemented the solution of capturing information from Databricks status page in real-time 24x7 and load that into a log or table...

https://learn.microsoft.com/en-us/azure/databricks/resources/status

what is the best way to be informed about Databricks availability? Just to create a webhook and get notified?

My requirement is to also report on the service downtimes, and device several KPIs out of it...

Appreciate the mindshare..

Re: Azure Databricks Status

TheRealOliver — Fri, 25 Apr 2025 15:44:50 GMT

It seems that the webhook is the way!

There is nothing about system status in Databricks REST API.

There is nothing about system status in the System Tables schema.

Re: Azure Databricks Status

noorbasha534 — Fri, 25 Apr 2025 16:11:20 GMT

@TheRealOliver thanks. Yes, but have you come across someone doing his, i mean listening to the status 365 days a year 24x7....with current mail subscription, we have alert fatigue as our mail box is also leveraged for other workloads also 🙂

Re: Azure Databricks Status

TheRealOliver — Fri, 25 Apr 2025 18:30:33 GMT

I agree that status updates must be useful and actionable.

If I understood your question correctly, you want to have the Databricks service notifications to be available in a Databricks table. I wouldn't use email subscription for this. I would use the webhook approach.

I am not proficient with Azure though so I can only describe the design in general terms:

Deploy a serverless HTTP endpoint (maybe with Azure Logic Apps or Azure Functions). The purpose of the endpoint would be to save incoming requests to blob storage.
Submit the endpoint URL to the status page and configure which notifications you want to receive.
Set up incremental ingestion from blob storage in Databricks and trigger it by a schedule at a desired frequency to collect the notifications from blob storage and save them to a Delta table.
Set up downstream processes that would trigger other actions in response to specific notifications.

I hope this helps!