I am getting this same issue. Occasionally a cell will display "Running Command" for as long as an hour. This can happen even for simple commands that ordinarily run in less than a second. I have tried restarting the cluster, attaching to a differen...
I have a similar issue. Several times per week I experience very slow (5 minutes +) of "running command" on a cell that should take sub 1 second to execute. It usually solves the problem to restart the cluster, but still a major inconvenience.
@cfregly @Vida Ha I'm having trouble with the same "task not serializable" error when calling foreachPartition.My code looks like: myDF
.foreachPartition { (rddpartition: Iterator[Row]) =>
val url = "jdbc:sqlserver://<myurl>"
val un = dbutils...
