01-06-2022 08:14 AM
Hi Team,
We have a job it completes in 3 minutes in one Databricks cluster, if we run the same job in another databricks cluster it is taking 3 hours to complete.
I am quite new to Databricks and need your guidance on how to find out where databricks spend a long-time during execution job. Any specific things I can check in Executor/driver logs?
Regards,
Rajesh.
01-31-2022 12:36 AM
Hi @Rajesh Kannan R , This article describes how to use monitoring dashboards to find performance bottlenecks in Spark jobs on Azure Databricks.
01-06-2022 10:31 AM
Hi @ RajeshRK! My name is Kaniz, and I'm the technical moderator here. Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question first. Or else I will get back to you soon. Thanks.
01-06-2022 09:54 PM
Hi Kaniz,
Sure, thanks for the response.
Regards,
Rajesh.
01-10-2022 08:04 AM
Hi @Rajesh Kannan R ,
For driver logs,
For executor logs, the process is a bit more involved:
01-10-2022 10:08 PM
Hi @Kaniz Fatma
I am able to extract the logs, but I need some pointer where the databricks spend a longer time during job execution.
01-31-2022 12:36 AM
Hi @Rajesh Kannan R , This article describes how to use monitoring dashboards to find performance bottlenecks in Spark jobs on Azure Databricks.
3 weeks ago
Hi @Kaniz ,
I am saving logs of my databricks Job Compute From ADF, How can i open those files that present in dbfs location.
Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections.
Click here to register and join today!
Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.