https://community.databricks.com/t5/technical-blog/ray-monitoring-made-easy-prometheus-amp-grafana-with-ray-on/ba-p/108641 <H2>Intro</H2> <P><SPAN>Ray is rapidly becoming the standard for logic-parallel computing, enabling many Databricks customers to accelerate a wide range of Python workloads. Since its </SPAN><A href="https://www.databricks.com/blog/announcing-general-availability-ray-databricks" target="_blank" rel="noopener"><SPAN>general availability on Databricks in early 2024</SPAN></A><SPAN>, Ray on Databricks has opened up new possibilities for cluster-based computing on the platform.</SPAN></P> <P><SPAN>Effective management of distributed cluster infrastructure relies heavily on observability and monitoring. The </SPAN><A href="https://docs.ray.io/en/latest/ray-observability/getting-started.html" target="_blank" rel="noopener"><SPAN>open-source Ray dashboard</SPAN></A><SPAN> is a vital tool for Ray developers, providing a web-based interface that visualizes system state, facilitates monitoring and debugging of Ray applications, and helps track performance and troubleshoot issues.</SPAN></P> <P><A href="https://docs.databricks.com/en/machine-learning/ray/start-ray.html#run-ray-on-databricks" target="_blank" rel="noopener"><SPAN>Launching</SPAN></A><SPAN> a Ray on Databricks cluster provides the basic features of the Ray dashboard, including job status, task details, and cluster node monitoring. In order to support hardware-level details for a running cluster that feeds into the </SPAN><A href="https://docs.ray.io/en/latest/ray-observability/getting-started.html#dash-metrics-view" target="_blank" rel="noopener"><SPAN>Metrics View</SPAN></A><SPAN>, this blog outlines a simple script to set up Prometheus and Grafana.</SPAN></P> <H2><SPAN>[Recommended] Prometheus+Grafana Setup on Databricks Ray cluster</SPAN></H2> <P><SPAN>The process described below can be used to setup a Prometheus+Grafana on an already-running Ray on Spark cluster. This is the most common way Ray is used on Databricks: either on an interactive All Purpose or dedicated Job cluster, a Ray cluster is initialized using the already-existing Spark context. A lightweight shell script is then used to&nbsp;start Prometheus+Grafana.</SPAN></P> <OL> <LI style="font-weight: 400;" aria-level="1"><SPAN>Review the init script available at this Github location: <A href="https://github.com/databricks-industry-solutions/ray-framework-on-databricks/tree/main/Ray_Dashboard_Metrics" target="_self">setup_monitoring.sh</A></SPAN><A href="https://github.com/databricks-industry-solutions/ray-framework-on-databricks/tree/main/Ray_Dashboard_Metrics" target="_self"><SPAN>&nbsp;</SPAN></A></LI> <UL> <LI style="font-weight: 400;" aria-level="2">Note: The init script is provided under the <A href="https://github.com/databricks-industry-solutions/scientific-computing-ray-on-spark/blob/main/LICENSE.md" target="_self">DB License</A>. Subject to the license terms, the code is provided as-is with no warranties and you may use it as-is or extend/customize it as needed.</LI> </UL> <LI>See the accompanying notebook for instructions on how to run via magic %sh command:&nbsp;prometheus-metrics-with-ray-dashboard.ipynb.</LI> <LI>After initializing the Ray on Spark cluster, run the uploaded/cloned script via <FONT face="courier new,courier">%sh ./setup_monitoring.sh</FONT>, then check out the Ray dashboard to see live cluster metrics! <UL> <LI>See this blog for best practices on cluster setup:&nbsp;<A href="https://community.databricks.com/t5/technical-blog/ray-on-spark-a-practical-architecture-and-setup-guide/ba-p/127511" target="_self">Ray on Spark: A Practical Architecture and Setup Guide</A></LI> </UL> </LI> </OL> <P>You can use this approach on a Global or Local Ray cluster, and adapt to any enterprise monitoring/logging standards at your organization. This lightweight process can also be used for large Ray on Databricks deployments: put the script centrally in a workspace or UC Volume, then use it from within many automated jobs that create their own or re-use an existing Ray on Spark cluster.&nbsp;</P> <H2><SPAN>[Legacy] Init Script-based Setup Process</SPAN></H2> <P>&nbsp;</P> <LI-SPOILER>NOTE: as of September 2025, Databricks no longer recommends using init scripts. The section below is preserved for reference only, but we recommend using the approach detailed above.&nbsp;</LI-SPOILER> <P>&nbsp;</P> <P><SPAN>The script below initializes a Global Ray Cluster that can be shared by multiple Ray applications and/or multiple users. Note the important steps below, including how developers should connect to the cluster with the </SPAN><FONT face="courier new,courier"><SPAN>ray.init()</SPAN></FONT><SPAN> command. Complete the following steps to use this approach:</SPAN></P> <OL> <LI style="font-weight: 400;" aria-level="1"><SPAN>Review the init script available at </SPAN><A href="https://github.com/databricks-industry-solutions/scientific-computing-ray-on-spark/blob/main/Ray_Dashboard_Metrics/init_ray_prometheus_grafana.sh" target="_blank" rel="noopener"><SPAN>this Github location</SPAN></A><SPAN>.&nbsp;</SPAN></LI> <UL> <LI style="font-weight: 400;" aria-level="2">Note: The init script is provided under the <A href="https://github.com/databricks-industry-solutions/scientific-computing-ray-on-spark/blob/main/LICENSE.md" target="_self">DB License</A>. Subject to the license terms, the code is provided as-is with no warranties and you may use it as-is or extend/customize it as needed.</LI> </UL> <LI><SPAN>Upload .sh init script to your Databricks workspace or UC Volume (<A href="https://docs.databricks.com/en/init-scripts/index.html" target="_blank" rel="noopener">init script docs</A>)</SPAN><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="tjcycyota_0-1738604153886.png" style="width: 775px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/14555i01E8A26020B7E727/image-dimensions/775x306?v=v2" width="775" height="306" role="button" title="tjcycyota_0-1738604153886.png" alt="tjcycyota_0-1738604153886.png" /></span></LI> <LI><SPAN><SPAN>Create a cluster to be used for Ray applications (see recommendations below)&nbsp;</SPAN></SPAN><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="tjcycyota_1-1738604203734.png" style="width: 775px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/14557i97DEB22B336B80BA/image-dimensions/775x370?v=v2" width="775" height="370" role="button" title="tjcycyota_1-1738604203734.png" alt="tjcycyota_1-1738604203734.png" /></span></LI> <LI><SPAN><SPAN>Under Advanced, add the init script from Step 1 using its full file path:&nbsp;</SPAN></SPAN><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="tjcycyota_2-1738604269261.png" style="width: 777px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/14558iF439B78ACAB5476E/image-dimensions/777x265?v=v2" width="777" height="265" role="button" title="tjcycyota_2-1738604269261.png" alt="tjcycyota_2-1738604269261.png" /></span></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>If you are using a multi-node Ray cluster, add the environment variable </SPAN><FONT face="courier new,courier"><SPAN>RAY_MAX_WORKERS=&lt;max workers&gt;</SPAN></FONT> <UL> <LI style="font-weight: 400;" aria-level="1"><SPAN><SPAN>Note: If using Spark+Ray together, review <A href="https://docs.databricks.com/en/machine-learning/ray/scale-ray.html#computation-resource-configuration-for-apache-spark-and-ray-hybrid-workloads" target="_blank" rel="noopener">best practices for resource sharing</A> and set <FONT face="courier new,courier">RAY_MAX_WORKERS</FONT> env variable to a number less than the total workers so Spark SQL tasks are not starved of resources.</SPAN></SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN><SPAN>&nbsp;</SPAN></SPAN><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="tjcycyota_3-1738604284216.png" style="width: 713px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/14559i0E8F97A80133CE6B/image-dimensions/713x549?v=v2" width="713" height="549" role="button" title="tjcycyota_3-1738604284216.png" alt="tjcycyota_3-1738604284216.png" /></span></LI> </UL> </LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Start the cluster. The cluster will take several extra minutes to start as the Ray on Spark cluster initializes and Prometheus+Grafana are set up.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Navigate to the Ray Dashboard. The easiest way to find this unique URL is:</SPAN> <UL> <LI style="font-weight: 400;" aria-level="2"><SPAN>On the cluster page, go to Apps tab → Web Terminal.&nbsp;</SPAN><SPAN>Copy the URL from the Web Terminal. It should look something like:&nbsp;</SPAN><FONT face="courier new,courier"><SPAN>https://…/driver-proxy/o/…/{cluster_id}/7681?o=...</SPAN></FONT></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Edit this link to point to the port where the Ray Dashboard is exposed. Replace the Web Terminal port number with 9999, as defined in the script:&nbsp;</SPAN><FONT face="courier new,courier"><SPAN>https://…/driver-proxy/o/…/{cluster_id}/</SPAN><STRONG>9999/</STRONG></FONT></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN><STRONG>Important</STRONG>: Make sure you include the trailing “/”</SPAN></LI> <LI><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="tjcycyota_4-1738604449452.png" style="width: 728px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/14560i16AC255FC0E3547E/image-dimensions/728x417?v=v2" width="728" height="417" role="button" title="tjcycyota_4-1738604449452.png" alt="tjcycyota_4-1738604449452.png" /></span> <P>&nbsp;</P> </LI> </UL> </LI> <LI style="font-weight: 400;" aria-level="1"><SPAN><SPAN>On the Metrics tab, you should now see Ray metrics provided by Prometheus+Grafana!&nbsp;</SPAN></SPAN><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="tjcycyota_5-1738604468013.png" style="width: 767px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/14561i967DE5A4A3D3DAC5/image-dimensions/767x328?v=v2" width="767" height="328" role="button" title="tjcycyota_5-1738604468013.png" alt="tjcycyota_5-1738604468013.png" /></span></LI> </OL> <P><SPAN>Now you can submit any Ray application to this cluster. In a Databricks notebook, connect to the cluster, then use the following technique to initialize the Ray session:</SPAN></P> <P>&nbsp;</P> <P>&nbsp;</P> <P>&nbsp;</P> <LI-CODE lang="python">import ray import os head_ip = os.getenv("SPARK_LOCAL_IP") print(f"Using Ray Head node at: {head_ip}") ray.init(address=f'ray://{head_ip}:10001')</LI-CODE> <P>&nbsp;</P> <P>&nbsp;</P> <P>&nbsp;</P> <P><SPAN>Note that </SPAN><STRONG>if you do not use the ray.init() technique above</STRONG><SPAN>, you will not be connecting to the Ray global cluster provisioned in the init script. The script has been developed exclusively for the purpose of sharing a global Ray cluster script between one or many applications, but can be adapted for single-user Ray use-cases.</SPAN></P> <P><SPAN>Additionally, with this process the Databricks cluster will </SPAN><STRONG>never automatically terminate</STRONG><SPAN>, even if you have selected autotermination settings in the Databricks UI. To avoid long-lived clusters, we recommend </SPAN><A href="https://docs.databricks.com/en/compute/clusters-manage.html#terminate-a-compute" target="_blank" rel="noopener"><SPAN>terminating the cluster</SPAN></A><SPAN> when work is complete or setting up an </SPAN><A href="https://docs.databricks.com/api/workspace/clusters/delete" target="_blank" rel="noopener"><SPAN>automation</SPAN></A><SPAN> to terminate all-purpose clusters at the end of the work period. As mentioned above, if you are using Spark and Ray together, review </SPAN><A href="https://docs.databricks.com/en/machine-learning/ray/scale-ray.html#computation-resource-configuration-for-apache-spark-and-ray-hybrid-workloads" target="_blank" rel="noopener"><SPAN>best practices for resource sharing</SPAN></A><SPAN> and ensure sufficient resources are available for both processes, which might require utilizing both the </SPAN><A href="https://docs.databricks.com/en/compute/troubleshooting/debugging-spark-ui.html" target="_blank" rel="noopener"><SPAN>Spark UI</SPAN></A><SPAN> and Ray Dashboard.</SPAN></P> <H2><SPAN>Environment Compatibility</SPAN></H2> <P><SPAN>This script has been tested under the following conditions:</SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><SPAN>Clouds: AWS Databricks, Azure Databricks</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Runtimes: 15.4 ML LTS, 16.1 ML&nbsp;</SPAN></LI> <UL> <LI style="font-weight: 400;" aria-level="1"><SPAN>This script may work on other Machine Learning runtimes with Ray pre-installed.</SPAN></LI> <LI style="font-weight: 400;" aria-level="2"><SPAN>Databricks Serverless compute does not allow init scripts.</SPAN></LI> </UL> <LI style="font-weight: 400;" aria-level="1"><SPAN>Cluster Access Mode: </SPAN><A href="https://docs.databricks.com/en/compute/access-mode-limitations.html" target="_blank" rel="noopener"><SPAN>Dedicated</SPAN></A></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Scaling: Single-node, Fixed-size multi-node, Autoscaling multi-node</SPAN></LI> </UL> <H2><SPAN>Debugging</SPAN></H2> <P><SPAN>In the case of errors or failure for cluster to start:</SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><A href="https://docs.databricks.com/en/init-scripts/logs.html#where-are-init-script-logs-written" target="_blank" rel="noopener"><SPAN>Review docs for enabling init script logs. </SPAN></A><SPAN>Enable cluster logs for the cluster, then start the cluster again.&nbsp;</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Navigate to these logs and fix any errors from the </SPAN><SPAN>…sh.stderr.log</SPAN><SPAN> file</SPAN></LI> </UL> <P><SPAN>In the case that the cluster starts, but the Ray dashboard is not showing metrics properly:</SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><SPAN>Go to the cluster → Apps → Web Terminal</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>In the Terminal, execute the following commands to review log files from cluster setup.&nbsp;</SPAN></LI> <UL> <LI style="font-weight: 400;" aria-level="2"><SPAN>Python log for Ray cluster setup: </SPAN> <FONT face="courier new,courier"><SPAN>cat /local_disk0/tmp/rayonsparkinit.log</SPAN></FONT></LI> <LI style="font-weight: 400;" aria-level="2"><SPAN>Prometheus setup log:</SPAN> <FONT face="courier new,courier"><SPAN>cat /local_disk0/tmp/prometheus.log</SPAN></FONT></LI> <LI style="font-weight: 400;" aria-level="2"><SPAN>Grafana setup log: </SPAN> <FONT face="courier new,courier"><SPAN>cat /local_disk0/tmp/grafana.log</SPAN><SPAN>&nbsp;</SPAN></FONT></LI> </UL> <LI style="font-weight: 400;" aria-level="1"><SPAN>Fix any errors in the init script or cluster setup based on these logs</SPAN></LI> </UL> <H2><SPAN>Resources</SPAN></H2> <UL> <LI style="font-weight: 400;" aria-level="1"><SPAN>[Github repo] </SPAN><A href="https://github.com/databricks-industry-solutions/scientific-computing-ray-on-spark/blob/main/Ray_Dashboard_Metrics/init_ray_prometheus_grafana.sh" target="_blank" rel="noopener"><SPAN>Scientific Computing Ray on Spark</SPAN></A></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>[Ray docs] </SPAN><A href="https://docs.ray.io/en/latest/ray-observability/getting-started.html#dash-metrics-view" target="_blank" rel="noopener"><SPAN>Ray Dashboard, metrics view&nbsp;</SPAN></A></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>[Ray docs] </SPAN><A href="https://docs.ray.io/en/latest/cluster/configure-manage-dashboard.html#observability-visualization-setup" target="_blank" rel="noopener"><SPAN>Embed Grafana visualizations into Ray Dashboard</SPAN></A></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>[Prometheus docs] </SPAN><A href="https://prometheus.io/docs/prometheus/latest/configuration/configuration/" target="_blank" rel="noopener"><SPAN>Prometheus configuration</SPAN></A></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>[Grafana docs] </SPAN><A href="https://grafana.com/docs/grafana/latest/setup-grafana/configure-grafana/" target="_blank" rel="noopener"><SPAN>Configure Grafana</SPAN></A></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>[Databricks docs] </SPAN><A href="https://docs.databricks.com/en/init-scripts/logs.html#where-are-init-script-logs-written" target="_blank" rel="noopener"><SPAN>Init script logging</SPAN></A></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>[Databricks docs] </SPAN><A href="https://docs.databricks.com/en/init-scripts/environment-variables.html" target="_blank" rel="noopener"><SPAN>Set and use environment variables with init scripts</SPAN></A></LI> <LI style="font-weight: 400;" aria-level="1">[Databricks docs] <A style="font-family: inherit; background-color: #ffffff;" href="https://docs.databricks.com/en/compute/access-mode-limitations.html" target="_blank" rel="noopener">Compute access mode limitations for Unity Catalog</A></LI> </UL> Tue, 14 Oct 2025 13:38:00 GMT tj-cycyota 2025-10-14T13:38:00Z https://community.databricks.com/t5/technical-blog/ray-monitoring-made-easy-prometheus-amp-grafana-with-ray-on/ba-p/108641 <P>This blog post provides a simple initialization script to set up Prometheus and Grafana on a Ray cluster on Databricks, enabling the Metrics View in the Ray dashboard for improved observability and monitoring. The script allows developers to track performance and troubleshoot issues in their Ray applications, enabling enhanced observability on the Databricks platform.</P> Tue, 14 Oct 2025 13:38:00 GMT https://community.databricks.com/t5/technical-blog/ray-monitoring-made-easy-prometheus-amp-grafana-with-ray-on/ba-p/108641 tj-cycyota 2025-10-14T13:38:00Z