topic Re: Need help to analyze databricks logs for a long-running job. in Data Engineering

Need help to analyze databricks logs for a long-running job.

RajeshRK — Thu, 06 Jan 2022 16:14:40 GMT

Hi Team,

We have a job it completes in 3 minutes in one Databricks cluster, if we run the same job in another databricks cluster it is taking 3 hours to complete.

I am quite new to Databricks and need your guidance on how to find out where databricks spend a long-time during execution job. Any specific things I can check in Executor/driver logs?

Regards,

Rajesh.

Re: Need help to analyze databricks logs for a long-running job.

RajeshRK — Fri, 07 Jan 2022 05:54:45 GMT

Hi Kaniz,

Sure, thanks for the response.

Regards,

Rajesh.

Re: Need help to analyze databricks logs for a long-running job.

RajeshRK — Tue, 11 Jan 2022 06:08:12 GMT

Hi @Kaniz Fatma

I am able to extract the logs, but I need some pointer where the databricks spend a longer time during job execution.

Re: Need help to analyze databricks logs for a long-running job.

AmitKP — Sun, 07 Apr 2024 19:53:50 GMT

Hi @Retired_mod ,

I am saving logs of my databricks Job Compute From ADF, How can i open those files that present in dbfs location.