cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Need help to analyze databricks logs for a long-running job.

RajeshRK
Contributor

Hi Team,

We have a job it completes in 3 minutes in one Databricks cluster, if we run the same job in another databricks cluster it is taking 3 hours to complete.

I am quite new to Databricks and need your guidance on how to find out where databricks spend a long-time during execution job. Any specific things I can check in Executor/driver logs?

Regards,

Rajesh.

1 ACCEPTED SOLUTION

Accepted Solutions

Kaniz
Community Manager
Community Manager

Hi @Rajesh Kannan R​ , This article describes how to use monitoring dashboards to find performance bottlenecks in Spark jobs on Azure Databricks.

View solution in original post

6 REPLIES 6

Kaniz
Community Manager
Community Manager

Hi @ RajeshRK! My name is Kaniz, and I'm the technical moderator here. Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question first. Or else I will get back to you soon. Thanks.

Hi Kaniz,

Sure, thanks for the response.

Regards,

Rajesh.

Kaniz
Community Manager
Community Manager

Hi @Rajesh Kannan R​ ,

For driver logs,

  1. Click on Jobs
  2. Click the job you want to see logs for
  3. Click "Logs".

For executor logs, the process is a bit more involved:

  1. Click on Clusters
  2. Choose the cluster in the list corresponding to the job
  3. Click Spark UI
  4. Now you have to choose the worker for which you want to see logs. Click the nodes list (it's on the far right, next to "Apps") and then you can click stdout or stderr to see the logs

Hi @Kaniz Fatma​ 

I am able to extract the logs, but I need some pointer where the databricks spend a longer time during job execution.

Kaniz
Community Manager
Community Manager

Hi @Rajesh Kannan R​ , This article describes how to use monitoring dashboards to find performance bottlenecks in Spark jobs on Azure Databricks.

AmitKP
New Contributor II

Hi @Kaniz ,

I am saving logs of my databricks Job Compute From ADF, How can i open those files that present in dbfs location.