cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Get Started Discussions
Start your journey with Databricks by joining discussions on getting started guides, tutorials, and introductory topics. Connect with beginners and experts alike to kickstart your Databricks experience.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

How do you analyze performance

Newbienewbster
New Contributor II

Curious to hear how you guys optimize compute. As in how you dig into the details of the Spark execution and improve?

1 REPLY 1

mhiltner
Databricks Employee
Databricks Employee

That is it. Usually, people take the time it takes to run a job/query/process as their KPI. 

Then you start to check which processes are taking more time, drilling down one by one. Sometimes it could be a misplaced .cache(), .collect() or display() that makes spark effectively calculate everything. You could also do the same for queries with the query profiler, checking whether there was shuffle, how many rows are being processed and whether there was disk spill. You can also check for skewness. 

I really like this blog: https://www.databricks.com/discover/pages/optimize-data-workloads-guide

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local communityโ€”sign up today to get started!

Sign Up Now