Stuck in "Running Command ..."

[Carneiro]

Hi,

Since yesterday, without a known reason, some commands that used to run daily are now stuck in a "Running command" state. Commands as:

dataframe.toPandas() 

dataframe.show(n=1) 

dataframe.description() 

dataframe.write.format("csv").save(location) 

get stuck in a “Running command …” state and I receive no output 

(or error) until the server stops for inactivity. In the best scenario, the command

runs after a long time and in the worse scenario, the job cannot complete due to 

inactivity. 

This is happening suddenly since yesterday and there were no updates on the 

Databricks runtime or in the cluster. I would appreciate any kind of advice. 

About the cluster:

61GB | 8 Cores | DBR 10.4 LTS | Spark 3.2.1 | Scala 2.12

Regards, Luiz

[jose_gonzalez · Accepted solution]

Hi @Luiz Carneiro​ ,

Could you split your Spark's actions into more CMD (paragraphs) and run one at a time to check where it could be taking the extra time. Also, Pandas only runs on your driver. Have you try to use Python or Scala APIs instead? in case you are familiar with Pandas APIs, then you can use Koalas. Docs here https://koalas.readthedocs.io/en/latest/

[Carneiro]

Hey Jose, thanks a lot for your reply.

I still did not understand completely what happened, but this week it seems that everything work again normally and I did no big changes. If I find something insightful, I will write It here. Thanks anyhow!