I am having a problem very similar.

Since yesterday, without a known reason, some commands that used to run daily are now stuck in a "Running command" state. Commands like:

dataframe.show(n=1)

dataframe.toPandas()

dataframe.description()

dataframe.write.format("csv").save(location)

are now stuck also for quite small dataframes with 28 rows and 5 columns, for example. I would appreciate any help since the problem is also in important daily jobs.