Need help to analyze databricks logs for a long-running job.

[RajeshRK]

Hi Team,

We have a job it completes in 3 minutes in one Databricks cluster, if we run the same job in another databricks cluster it is taking 3 hours to complete.

I am quite new to Databricks and need your guidance on how to find out where databricks spend a long-time during execution job. Any specific things I can check in Executor/driver logs?

Regards,

Rajesh.

[RajeshRK]

Hi Kaniz,

Sure, thanks for the response.

Regards,

Rajesh.

[RajeshRK]

Hi @Kaniz Fatma​ 

I am able to extract the logs, but I need some pointer where the databricks spend a longer time during job execution.

[AmitKP]

Hi @Retired_mod ,

I am saving logs of my databricks Job Compute From ADF, How can i open those files that present in dbfs location.