Running Command is often stuck on "Running Command..."

[Retko]

Hi,

when running command, it often gets stuck and message below it says: "Running Command..."

What can I do with it besides of restarting cluster?

Also tried reattaching and clearing state, but no help here.

Thanks

[Vinay_M_R]

When a command gets stuck and displays the message "Running Command...", it could be due to several reasons such as a resource-intensive task, network issues, or a problem with the cluster. Here are some steps you can take to troubleshoot this issue:

1. Check the Cluster's Status and Logs: Use the Databricks UI to check the status of your cluster. If the cluster is in a "Terminated" state, you will need to restart it. You can also check the logs for any error messages or warnings.

2. Check the Resource Usage: If your command is resource-intensive, it might be taking longer to execute. You can check the Spark UI to see the resource usage of your command. If necessary, you can increase the resources of your cluster.

3. Try Running a Simple Command: Try running a simple command to see if it executes successfully. If it does, then the issue might be with the specific command you're trying to run.

4. Check Network Issues: If you're running Databricks in a cloud environment, there could be network issues that are causing the command to get stuck. Check your network connectivity and firewall settings.

Please provide me with more details about the command you're trying to run and the environment you're running it in.

[Lakshay]

Hi, Could you please help us understand what sort of query you are running and what is the cluster configuration you are using? Also, check for any issues in the logs which can give an idea on why the command is getting stuck.

[Tharun-Kumar]

@Retko 

If you are only seeing Running Command and no spark job information, then the driver is performing some operation and it's hanging there. One of the common scenario, this could happen is when you have too many small files in your input directory. Listing will be performed on the driver and it would take much time to list all the small files.

If you are able to see Spark Jobs below running command, then spark jobs have been triggered at the executor side and the executors are processing your data. In this case, you could traverse to the SPARK UI, look at the task level and SQL information to understand the volume of data you are processing and how it is getting processed.

[Debayan]

Hi, do you see this while running a command in the notebook? 
Please tag @Debayan  with your next comment which will notify me. Thanks!