How do you analyze performance

Newbienewbster — Thu, 13 Jun 2024 20:13:01 GMT

Curious to hear how you guys optimize compute. As in how you dig into the details of the Spark execution and improve?

Re: How do you analyze performance

mhiltner — Fri, 14 Jun 2024 10:39:08 GMT

That is it. Usually, people take the time it takes to run a job/query/process as their KPI.

Then you start to check which processes are taking more time, drilling down one by one. Sometimes it could be a misplaced .cache(), .collect() or display() that makes spark effectively calculate everything. You could also do the same for queries with the query profiler, checking whether there was shuffle, how many rows are being processed and whether there was disk spill. You can also check for skewness.

I really like this blog: https://www.databricks.com/discover/pages/optimize-data-workloads-guide

topic How do you analyze performance in Get Started Discussions

How do you analyze performance

Re: How do you analyze performance