Hi,

How are you doing today? , As per my understanding, It sounds like you need a centralized observability solution that works across multiple Databricks workspaces, supports serverless compute, and provides alerts and detailed logs. Since webhooks and Databricks system tables have limitations, you might consider Databricks Audit Logs + a centralized monitoring dashboard. Audit logs (enabled via AWS CloudTrail or Azure Monitor) can capture job status across all workspaces, and you can process them using a Lakehouse approach—storing logs in a Delta table and querying them with Databricks SQL. For real-time monitoring, you could set up a Databricks Job that periodically aggregates job statuses and pushes them to a tool like Prometheus, Grafana, or a custom dashboard. To get alerts, consider Databricks Alerts (SQL Alerts), CloudWatch (AWS), or Azure Monitor with email notifications. If you want deeper visibility, third-party tools like Monte Carlo, Acceldata, or Unravel offer better observability and cost insights. Let me know if you want help setting up one of these solutions!

Regards,

Brahma