cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Checking spark performance locally

mghildiy
New Contributor

I am experimenting with spark, on my local machine. So, is there some tool/api available to check the performance of the code I write?

For eg. I write:

val startTime = System.nanoTime()
  invoicesDF
    .select(
      count("*").as("Total Number Of Invoices"),
      sum("Quantity").as("Total Quantity"),
      avg("UnitPrice").as("Avg Unit Price"),
      countDistinct("InvoiceNo").as("Number Of Unique Invoices")
    )
    .show()
 
  println((System.nanoTime() - startTime) / 1000000000.0)

I am using System.nanoTime to do these calculations, but am not sure of this is correct way.

Btw, this takes time of around 2.7 seconds, using 3 threads, and file having 541900 records. Is it a good performance? It is a 16GB, 4 core, 2.40GHz Intel processor.

2 REPLIES 2

Hubert-Dudek
Esteemed Contributor III

Please check the details about your code (task in jobs) in Spark UI.

Kaniz
Community Manager
Community Manager

Hi @mghildiy​ ​, We haven’t heard from you since the last response from @Hubert Dudek​, and I was checking back to see if you have a resolution yet.

If you have any solution, please share it with the community as it can be helpful to others. Otherwise, we will respond with more details and try to help.

Also, Please don't forget to click on the "Select As Best" button whenever the information provided helps resolve your question.

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.