Yes, I face the same problem. The only way I could overcome this is my disabling auto-scaling, setting worker = 0. So I will end up having just one driver node and will not be able to test the parallel execution functionality.

This is the only way works for me. Thank you!

I have a Free Azure student account and I was facing similar challenges. This is how I solved mine after many hours of trial and error.

Mine is working now.

When creating the Azure Databricks resource group: Select Premium.

Don't use the 14 day trial.

Launch the Databricks workspace

When creating the cluster: Select Single Node under cluster mode

Select F4s with 4 core.

it should work for you now.

Hey,

Not a chance to choose 'Premium', the only option is 'Free Trial' 😞