Super slow SQL queries on an HC cluster

brickster_2018
Databricks Employee
Databricks Employee

I have a high concurrency cluster where multiple users are running. However, I see the queries are running very slow. I did debug the logs and see more time is spent on the Spark driver. on the Spark UI, I do not see slowness.