Hey , @darek554 

I'd bet this is just the instance being too small. m4.large is 2 cores / 8 GB, and on a single node the driver and executor have to share that  usually the executor never actually comes up, so your command just sits there waiting for resources. That's also why SQL doesn't work at all, since every SQL query needs an executor, while Python can sometimes limp through once the session finally wakes up.

Quick way to confirm: while something is hanging, open the Spark UI and check the Executors tab. If it's empty, that's your answer.

Try switching to an m5.xlarge or i3.xlarge and see if it goes away. m4 is also an older generation type that some workspaces don't really support anymore. If that doesn't fix it, have a look at the cluster Event Log  it usually says something useful about workers not attaching.