Ray dashboard no longer available

[Fed]

Has anyone else experienced the lack of access to the Ray dashboard since this week?

Last week worked fine.

rom ray.util.spark import setup_ray_cluster
 
setup_ray_cluster(...)

This used to output an HTML block with a link to the dashboard.

I can manually get the link with this, but the page shows 502 Bad Gateway

from ray.util.spark.cluster_init import _setup_ray_cluster
 
cluster = _setup_ray_cluster(...)
cluster.wait_until_ready() # this won't output the link because ray_ctx.address_info.get("webui_url", None) is falsy
cluster.start_hook.on_ray_dashboard_created(cluster.ray_dashboard_port)

[Anonymous]

@Federico Trifoglio​ :

There have been some recent reports of issues with accessing the Ray dashboard, particularly with the 502 Bad Gateway error. This could be due to a number of reasons, such as server overload, network issues, or bugs in the Ray software.

One thing you can try is to check if there are any known issues with the version of Ray you are using, and if there are, try upgrading to a newer version that might have addressed the issue.

Another thing you can try is to check the logs for any errors or warnings related to the Ray dashboard, and see if there are any clues as to what might be causing the issue.

You could also try reaching out to the Ray community for help, as they might have more information or suggestions for how to troubleshoot the issue.

[Fed]

I'm using the latest version of Ray (2.3.0) and I don't think it's a Ray issue because it worked perfectly fine until last week. It literally just happened on Monday, and then again everyday over the last 4 days.

I did open an issue a few days ago on GitHub

https://github.com/ray-project/ray/issues/33521

I also don't think it's a Ray issue because if I start ray (ray start --head) on my local machine I can access the dashboard via 127.0.0.1:8265

This is a minimal example of starting a ray cluster similar to based on what ray.util.spark.setup_ray_cluster seems to do behind the scenes

from ray.util.spark.utils import get_random_unused_port
import socket
 
ray_head_ip = socket.gethostbyname(spark.conf.get("spark.driver.host"))
ray_head_port = get_random_unused_port(ray_head_ip, min_port=9000, max_port=10000)
ray_dashboard_port = get_random_unused_port(
    ray_head_ip, min_port=9000, max_port=10000, exclude_list=[ray_head_port]
)
ray_dashboard_agent_port = get_random_unused_port(
    ray_head_ip,
    min_port=9000,
    max_port=10000,
    exclude_list=[ray_head_port, ray_dashboard_port],
)
dashboard_options = [
    "--dashboard-host=0.0.0.0",
    f"--dashboard-port={ray_dashboard_port}",
    f"--dashboard-agent-listen-port={ray_dashboard_agent_port}",
]
 
print("ray start --head" + f" --node-ip-address={ray_head_ip} --port={ray_head_port} " + " ".join(dashboard_options) + " --verbose")

If I then run the command with

%sh
 
ray start --head --node-ip-address=192.168.185.148 --port=9921 --dashboard-host=0.0.0.0 --dashboard-port=9686 --dashboard-agent-listen-port=9032 --verbose

I get the usual output (and no errors about the dashboard, not even in the driver logs) that the cluster has been started and I can confirm that the Ray head node port 9921 is listening

%sh
 
netstat -tulpn | grep :9921

But if I do the same for the dashboard port I don't get anything.

%sh
 
netstat -tulpn | grep :9686

The dashboard port is not listening.

[Anonymous]

Hi @Federico Trifoglio​ 

I'm sorry you could not find a solution to your problem in the answers provided.

Our community strives to provide helpful and accurate information, but sometimes an immediate solution may only be available for some issues.

I suggest providing more information about your problem, such as specific error messages, error logs or details about the steps you have taken. This can help our community members better understand the issue and provide more targeted solutions.

Alternatively, you can consider contacting the support team for your product or service. They may be able to provide additional assistance or escalate the issue to the appropriate section for further investigation.

Thank you for your patience and understanding, and please let us know if there is anything else we can do to assist you.

[Fed · Accepted solution]

The reason for the missing dashboard for me was due ​to not having installed some required dependencies. Shout out to the Ray community for their help.

I've submitted a PR (now merged) to add a warning message​ when such dependencies are missing.