any open-source library or init script for exporting Databricks job metrics to Prometheus

[Kirankumarbs]

Hello Community,

I’m exploring ways to export the default metrics from Databricks Jobs/Workflows to Prometheus and visualize them in Grafana.

There’s a nice option to use Databricks as a data source in Grafana, but that’s available only in the Enterprise edition. Before building a custom script or embedding logic directly into my Spark jobs, I’d like to check if there’s an open-source alternative.

Are there any recommended open-source tools, libraries, or init scripts that can help with this?

[Raman_Unifeye · Accepted solution]

@Kirankumarbs 

If you jst want to track Job status Success/Failure, Run Duration, and Costs, use exporter that talks to the Databricks Workspace API or System Tables.

grafana/databricks-prometheus-exporter - This is an open-source exporter. It queries Databricks System Tables  to export metrics on billing, job runs, and SQL Warehouse performance. it needs a Service Principal with access to system.lakeflow

[Kirankumarbs]

I checked library documentation! It seems good start for me to visualize System tables at Grafana!