https://community.databricks.com/t5/technical-blog/databricks-observability-using-grafana-and-prometheus/ba-p/96849 <H1><SPAN>Databricks Observability using Grafana and Prometheus</SPAN></H1> <P><SPAN>As software systems scale, the amount of data they process and the work they do grows with them.</SPAN> <SPAN>This is not limited only to cloud-native services but also data pipelines that support them. When you have just a handful of data jobs and pipelines to worry about it’s not too difficult to reason about their metrics, however as the number of those pipelines increases over time it’s important to keep a pulse on them. That’s where observability comes in. Setting up a good observability foundation for the future can be essential for the cost, performance, and correctness of your pipelines.</SPAN></P> <H2><SPAN>What is Observability?</SPAN></H2> <P><SPAN>Originally the term Observability was defined to mean examining the state of the system based on the data it generates.&nbsp;</SPAN><SPAN>Today, with the growth of cloud computing, it expanded to include different types of telemetry data - such as Logs, Metrics, and Traces (aka. the&nbsp;<A href="https://www.oreilly.com/library/view/distributed-systems-observability/9781492033431/ch04.html" target="_blank" rel="noopener">Three Pillars of Observability</A>).</SPAN><SPAN>&nbsp;</SPAN></P> <P><SPAN>In this blog post, we’ll focus on collecting and processing various metrics from your Databricks Spark jobs using free, open source tools such as </SPAN><A href="https://prometheus.io/" target="_blank" rel="noopener"><SPAN>Prometheus</SPAN></A><SPAN> and </SPAN><A href="https://grafana.com/" target="_blank" rel="noopener"><SPAN>Grafana</SPAN></A><SPAN>. In the next part, we’ll explore APM (Application Performance Monitoring) and explore runtime performance observability using </SPAN><A href="https://pyroscope.io/" target="_blank" rel="noopener"><SPAN>Apache Pyroscope</SPAN></A><SPAN>.</SPAN></P> <P><SPAN>All source code in this post can be found in the </SPAN><A href="https://github.com/rayalex/spark-databricks-observability-demo" target="_blank" rel="noopener"><SPAN>Spark/Databricks Observability Demo</SPAN></A><SPAN> on GitHub. It includes the necessary Terraform code to set up monitoring on the Databricks side, Docker Compose to spin up Grafana/Prometheus/Pyroscope, and other tools to help you get started.</SPAN></P> <H2><SPAN>Why it matters?</SPAN></H2> <P><SPAN>Observability is typically one of the </SPAN><I><SPAN>-ilities</SPAN></I><SPAN>, as in - </SPAN><A href="https://en.wikipedia.org/wiki/Non-functional_requirement" target="_blank" rel="noopener"><SPAN>Non-functional requirements</SPAN></A><SPAN> which are used to judge the operation of a system as a whole rather than in specific behaviors. So what does that mean for you, a </SPAN><I><SPAN>Software, DevSecOps, or a Data engineer </SPAN></I><SPAN>reading this? Well, a few things. Have you ever considered topics like:</SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><SPAN>How long do my jobs run? Are they subject to seasonality and require different parameters on weekends than on workdays?</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Am I overprovisioning my Spark clusters? Are my jobs autoscaling to tens of nodes, but due to IO constraints, I only use 50% of their CPU capacity?</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Are my business metrics correct? Does the number of input rows in </SPAN><I><SPAN>Table A</SPAN></I><SPAN> match the number of rows written in </SPAN><I><SPAN>Table B</SPAN></I><SPAN>?</SPAN></LI> </UL> <P><SPAN>And the most important - </SPAN><I><SPAN>how can I track these metrics over time and raise alerts if needed?</SPAN></I><I></I></P> <H2><SPAN>Enter Distributed Monitoring</SPAN></H2> <P><SPAN>Metrics are numerical representations of the health (and other aspects) of the system. Typically, we would collect CPU and RAM usage/utilization, memory consumption, network traffic, etc. In the world of Apache Spark™, this extends to JVM and performance utilization of Spark applications. We can capture information about JVM Heap usage, metrics about Spark Jobs and Tasks, scheduler, shuffle, etc.</SPAN></P> <P><SPAN>Spark Applications fit in quite well into this vision - due to its nature as a distributed system, we should be able to easily aggregate metrics across our workloads and derive insights from them.</SPAN></P> <H1><SPAN>The Tools</SPAN></H1> <P><SPAN>There are many tools out there that can be used to collect, aggregate, and visualize metrics. Here we’ll focus on two popular yet free options - Prometheus and Grafana. Prometheus is used to collect metrics from various sources (e.g. both hardware and software metrics) and Grafana is mainly used to visualize them and build views and dashboards that show a bigger picture on top.&nbsp;</SPAN></P> <P><SPAN>Spark actually comes with </SPAN><A href="https://spark.apache.org/docs/latest/monitoring.html#executor-metrics" target="_blank" rel="noopener"><SPAN>out-of-the-box support</SPAN></A><SPAN> for Prometheus. However, it doesn’t cover all the metrics, only a subset of them, and it’s not really suitable for short-lived jobs, as we’ll see.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="observability_architecture.jpg" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/12412i96801927009571C5/image-size/large?v=v2&amp;px=999" role="button" title="observability_architecture.jpg" alt="observability_architecture.jpg" /></span></P> <H2><SPAN>Prometheus Pushgateway</SPAN></H2> <P><SPAN>Prometheus is a pull-based metrics collector - that is, you have to configure it to periodically scrape certain metrics from an accessible endpoint. This works well for long-running services but it can be difficult to work around in case you have to monitor a lot of ephemeral workloads, such as one-time or periodic Jobs in Databricks.&nbsp;</SPAN></P> <P><SPAN>Here is where </SPAN><A href="https://github.com/prometheus/pushgateway" target="_blank" rel="noopener"><SPAN>Prometheus Pushgateway</SPAN></A><SPAN> comes into the picture. Pushgateway exists so that short-lived jobs can push their metrics to it, and we can configure Prometheus to target Pushgateway as a source. The Pushgateway itself will transparently cache and proxy the metrics pushed to it. We can leverage that behavior by deploying the Pushgateway alongside our Prometheus instance and configuring our Databricks Job Clusters to push metrics to Pushgateway instead.</SPAN></P> <H1><SPAN>Setup</SPAN></H1> <H2><SPAN>Setting up Databricks clusters with spark-metrics</SPAN></H2> <P><SPAN>Here, we’re using another library called spark-metrics. Internally, it integrates with Spark’s own observability model and pushes those metrics to Pushgateway. You can find an up-to-date form that works with Spark 3.5+ in the </SPAN><A href="https://github.com/rayalex/spark-metrics" target="_blank" rel="noopener"><SPAN>rayalex/spark-metrics</SPAN></A><SPAN> fork, as well as pre-built binaries.</SPAN></P> <P><SPAN>The easiest way to get going is to have an </SPAN><A href="https://docs.databricks.com/en/init-scripts/cluster-scoped.html" target="_blank" rel="noopener"><SPAN>init-script</SPAN></A><SPAN> that will install the necessary libraries and configure Pusgateway properly and in a reproducible way. We leverage init-scripts here to make sure the environment is ready before the Spark JVM process boots and configuration is picked up on startup.</SPAN></P> <P><SPAN>The </SPAN><A href="https://github.com/rayalex/spark-databricks-observability-demo/blob/main/main.tf#L18-L26" target="_blank" rel="noopener"><SPAN>accompanying Terraform example</SPAN></A><SPAN> will do these parts for you.</SPAN></P> <H3><SPAN>Setting up libraries</SPAN></H3> <P><SPAN>In order to hook into Spark’s JMX metrics - we need a compatible client that’s getting bootstrapped at runtime. As we mentioned earlier we’ll be using the </SPAN><A href="https://github.com/rayalex/spark-metrics" target="_blank" rel="noopener"><SPAN>spark-metrics</SPAN></A><SPAN> library for this. There are a few ways of doing this, but for the sake of simplicity we’ll be placing the jar in our </SPAN><A href="https://docs.databricks.com/en/files/workspace.html" target="_blank" rel="noopener"><SPAN>Workspace Files</SPAN></A><SPAN>.</SPAN></P> <P><SPAN>Finally, we can just copy the jar to its target location as part of our init-script:</SPAN></P> <P>&nbsp;</P> <LI-CODE lang="csharp">cp /Workspace/Users/.../spark_metrics.jar /databricks/jars</LI-CODE> <P>&nbsp;</P> <H3><SPAN>Configuring Pushgateway</SPAN></H3> <P><SPAN>For the library to be able to actually do anything, we need to create its configuration. This will include things like Pushgateway endpoint, which metrics to send, job names, etc.</SPAN></P> <P><SPAN>We’ll only show relevant parts here. However, you can look at the entire configuration for the full list of options. To make the script reusable, we’ll also use cluster environment variables to pull in some additional </SPAN><A href="https://github.com/rayalex/spark-databricks-observability-demo/blob/main/init-prometheus.sh#L16" target="_blank" rel="noopener"><SPAN>configuration options</SPAN></A><SPAN>.</SPAN></P> <P>&nbsp;</P> <LI-CODE lang="python"># configure spark metrics to use pushgateway as target pushgatewayHost=$PROMETHEUS_HOST jobName=$PROMETHEUS_JOB_NAME cat &gt;&gt; /databricks/spark/conf/metrics.properties &lt;&lt;EOL # Enable Prometheus for all instances by class name *.sink.prometheus.class=org.apache.spark.banzaicloud.metrics.sink.PrometheusSink # Prometheus pushgateway address *.sink.prometheus.pushgateway-address-protocol=http *.sink.prometheus.pushgateway-address=$pushgatewayHost *.sink.prometheus.period=5 *.sink.prometheus.labels=job_name=$jobName # Enable HostName in Instance instead of Appid (Default value is false i.e. instance=${appid}) *.sink.prometheus.enable-hostname-in-instance=true # Enable JVM metrics source for all instances by class name *.sink.jmx.class=org.apache.spark.metrics.sink.JmxSink *.source.jvm.class=org.apache.spark.metrics.source.JvmSource EOL</LI-CODE> <P>&nbsp;</P> <H3><SPAN>Configuring the Spark Clusters</SPAN></H3> <P><SPAN>The last step is to add the necessary configuration to the cluster. We’ve made it simpler to reuse the above script by injecting the configuration through your cluster’s environment variables. Currently, we only supply the job name as a metric tag, but you can expand this to fit your needs—e.g., adding specific IDs that are used internally, job version, run date, etc. Databricks itself will also supply a few metrics here, such as Databricks job run ID.</SPAN></P> <P>&nbsp;</P> <LI-CODE lang="ruby">PROMETHEUS_HOST=10.10.10.0:9091 PROMETHEUS_JOB_NAME=dbx-demo-job</LI-CODE> <P>&nbsp;</P> <P><SPAN>We also need to attach the init script to our interactive (or job) clusters. The </SPAN><A href="https://github.com/rayalex/spark-databricks-observability-demo/blob/main/main.tf#L29" target="_blank" rel="noopener"><SPAN>Terraform demo</SPAN></A><SPAN> we’re using will configure all of this for us.</SPAN></P> <P><SPAN>And that’s it! Running our job now will provision a cluster, load the new jar into the classpath and if everything goes well, read the configuration before spinning up the Spark Context. Next, let’s set up the rest of the tools and see this in action!</SPAN></P> <H2><SPAN>Setting up Prometheus, Pushgateway, and Grafana</SPAN></H2> <P><SPAN>The final piece of the puzzle is to set up Prometheus itself (alongside Pushgateway) and Grafana (to visualize the results). Our example comes with an already ready-to-go </SPAN><A href="https://github.com/rayalex/spark-databricks-observability-demo/blob/main/docker/docker-compose.yml" target="_blank" rel="noopener"><SPAN>docker-compose configuration</SPAN></A><SPAN> that will deploy everything for you in your environment of choice—the only requirement is connectivity between your Databricks Cluster and the VM that’s running Pushgateway. If you’re just setting up your workspace, I recommend setting it up with </SPAN><A href="https://learn.microsoft.com/en-us/azure/databricks/security/network/classic/vnet-inject" target="_blank" rel="noopener"><SPAN>VNet/VPC injection</SPAN></A><SPAN> to make sure you have enough flexibility to deploy your networking as needed.</SPAN></P> <P><SPAN>Let’s go through the components step by step. In the production system, you would probably deploy these components more resiliently and permanently, but for our example, it’s sufficient to use Docker Compose to demonstrate that it works.</SPAN></P> <H3><SPAN>Pushgateway</SPAN></H3> <P><SPAN>Pushgateway doesn’t require any specific configuration. You can just run it as-is, and it will listen to metrics on port 9091.</SPAN></P> <P>&nbsp;</P> <LI-CODE lang="python">pushgateway: image: prom/pushgateway command: --web.enable-admin-api ports: - '9091:9091'</LI-CODE> <P>&nbsp;</P> <H3><SPAN>Prometheus</SPAN></H3> <P><SPAN>Prometheus requires some configuration to work. Most importantly, we have to tell it to scrape metrics residing in Pushgateway occasionally. In our compose example, we provide the configuration as a mounted volume.</SPAN></P> <P>&nbsp;</P> <LI-CODE lang="python">prometheus: image: prom/prometheus ports: - '9090:9090' volumes: - ./config/prometheus.yml:/etc/prometheus/prometheus.yml command: --config.file=/etc/prometheus/prometheus.yml</LI-CODE> <P>&nbsp;</P> <P><SPAN>And the configuration file itself:</SPAN></P> <P>&nbsp;</P> <LI-CODE lang="python">global: scrape_interval: 5s evaluation_interval: 5s scrape_configs: - job_name: 'pushgateway' honor_labels: true honor_timestamps: true metrics_path: '/metrics' static_configs: - targets: ['pushgateway:9091']</LI-CODE> <P>&nbsp;</P> <P><SPAN>Here we’re telling Prometheus to scrape our (docker-compose) Pushgateway endpoint every 5 seconds.</SPAN></P> <H3><SPAN>Grafana</SPAN></H3> <P><SPAN>Once we have our metrics ingested, the last step is to explore and visualize them. Grafana also requires very little configuration, and it’s easy to extend it to consume metrics from Prometheus if you’re not already using it.&nbsp;</SPAN></P> <P><SPAN>Docker Compose configuration is fairly simple:</SPAN></P> <P>&nbsp;</P> <LI-CODE lang="python"> grafana: image: grafana/grafana ports: - '3000:3000' volumes: - ./config/grafana/datasources:/etc/grafana/provisioning/datasources/</LI-CODE> <P>&nbsp;</P> <P><SPAN>We can add the Prometheus Datasource through the UI, or programmatically as we’re doing here:</SPAN></P> <P>&nbsp;</P> <LI-CODE lang="python">apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://prometheus:9090 editable: true</LI-CODE> <P>&nbsp;</P> <P><SPAN>And we’re done. Next time our job runs (and in the case of interactive clusters, while they’re up), we should see the metrics show up in Grafana. Let’s explore a few of the examples.</SPAN></P> <H3><SPAN>Examples</SPAN></H3> <P><SPAN>A basic dashboard that shows information about simple PiCalculator Databrick Job, which runs every 5 minutes. You can see the number of executors, their average memory usage, GC statistics, and overall Spark tasks.&nbsp;</SPAN></P> <P><SPAN><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="prometheus_dashboard.jpg" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/12413i26979EC53E927CA0/image-size/large?v=v2&amp;px=999" role="button" title="prometheus_dashboard.jpg" alt="prometheus_dashboard.jpg" /></span></SPAN></P> <H1><SPAN>Conclusion</SPAN></H1> <P><SPAN>We’ve seen how we can easily and effectively gain insights into our Databricks applications metrics, using free and open source tools. In the next part, we’ll be exploring </SPAN><A href="https://en.wikipedia.org/wiki/Application_performance_management" target="_blank" rel="noopener"><SPAN>Application Performance Monitoring</SPAN></A><SPAN> with </SPAN><A href="https://pyroscope.io/" target="_blank" rel="noopener"><SPAN>Pyroscope</SPAN></A><SPAN> and show how we can dive deep into Spark internals and discover how can we track application performance in detail and monitor how it changes over time. Stay tuned!</SPAN></P> Wed, 30 Oct 2024 16:09:08 GMT aleksandar 2024-10-30T16:09:08Z https://community.databricks.com/t5/technical-blog/databricks-observability-using-grafana-and-prometheus/ba-p/96849 <P><span class="lia-inline-image-display-wrapper lia-image-align-left" image-alt="prometheus_dashboard.jpg" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/12411i9C8D5F8B93C92D48/image-size/large?v=v2&amp;px=999" role="button" title="prometheus_dashboard.jpg" alt="prometheus_dashboard.jpg" /></span></P> Wed, 30 Oct 2024 16:09:08 GMT https://community.databricks.com/t5/technical-blog/databricks-observability-using-grafana-and-prometheus/ba-p/96849 aleksandar 2024-10-30T16:09:08Z https://community.databricks.com/t5/technical-blog/databricks-observability-using-grafana-and-prometheus/bc-p/97013#M352 <P>Thanks for sharing these&nbsp;<a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/56643">@aleksandar</a>&nbsp;! Really insightful.</P> Thu, 31 Oct 2024 15:34:10 GMT https://community.databricks.com/t5/technical-blog/databricks-observability-using-grafana-and-prometheus/bc-p/97013#M352 NandiniN 2024-10-31T15:34:10Z https://community.databricks.com/t5/technical-blog/databricks-observability-using-grafana-and-prometheus/bc-p/100807#M397 <P>Thank you <a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/56643">@aleksandar</a>&nbsp;. Can you provide the link for Grafana dashboard file</P> Tue, 03 Dec 2024 17:11:09 GMT https://community.databricks.com/t5/technical-blog/databricks-observability-using-grafana-and-prometheus/bc-p/100807#M397 devron 2024-12-03T17:11:09Z https://community.databricks.com/t5/technical-blog/databricks-observability-using-grafana-and-prometheus/bc-p/128339#M723 <P>Thanks for sharing. Are serverless clusters supported as well?&nbsp;</P><P>&nbsp;</P> Wed, 13 Aug 2025 13:08:15 GMT https://community.databricks.com/t5/technical-blog/databricks-observability-using-grafana-and-prometheus/bc-p/128339#M723 sunnyjaindbrx 2025-08-13T13:08:15Z https://community.databricks.com/t5/technical-blog/databricks-observability-using-grafana-and-prometheus/bc-p/128343#M724 <P><A style="background-color: #ffffff;" target="_blank" rel="noopener">@sunnyjaindbrx</A><SPAN>&nbsp;It's not possible to do this on Serverless at the moment, as it would require running untrusted code/agents, which is not supported.</SPAN>&nbsp;</P> Wed, 13 Aug 2025 13:21:25 GMT https://community.databricks.com/t5/technical-blog/databricks-observability-using-grafana-and-prometheus/bc-p/128343#M724 aleksandar 2025-08-13T13:21:25Z https://community.databricks.com/t5/technical-blog/databricks-observability-using-grafana-and-prometheus/bc-p/128467#M728 <P>Okay thanks for getting back quickly. Do you know if there is any roadmap to have observability for serverless compute resource monitoring. Or emit the serverless cluster logs to external system via mechanism.&nbsp;</P> Thu, 14 Aug 2025 13:28:53 GMT https://community.databricks.com/t5/technical-blog/databricks-observability-using-grafana-and-prometheus/bc-p/128467#M728 sunnyjaindbrx 2025-08-14T13:28:53Z https://community.databricks.com/t5/technical-blog/databricks-observability-using-grafana-and-prometheus/bc-p/131732#M761 <P>Can I use this to export logs of a particular task and setup alerts based on logging message ?</P> Fri, 12 Sep 2025 07:09:22 GMT https://community.databricks.com/t5/technical-blog/databricks-observability-using-grafana-and-prometheus/bc-p/131732#M761 vasu2652 2025-09-12T07:09:22Z