โ01-06-2022 08:14 AM
Hi Team,
We have a job it completes in 3 minutes in one Databricks cluster, if we run the same job in another databricks cluster it is taking 3 hours to complete.
I am quite new to Databricks and need your guidance on how to find out where databricks spend a long-time during execution job. Any specific things I can check in Executor/driver logs?
Regards,
Rajesh.
โ01-31-2022 12:36 AM
Hi @Rajesh Kannan Rโ , This article describes how to use monitoring dashboards to find performance bottlenecks in Spark jobs on Azure Databricks.
โ01-06-2022 10:31 AM
Hi @ RajeshRK! My name is Kaniz, and I'm the technical moderator here. Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question first. Or else I will get back to you soon. Thanks.
โ01-06-2022 09:54 PM
Hi Kaniz,
Sure, thanks for the response.
Regards,
Rajesh.
โ01-10-2022 08:04 AM
Hi @Rajesh Kannan Rโ ,
For driver logs,
For executor logs, the process is a bit more involved:
โ01-10-2022 10:08 PM
Hi @Kaniz Fatmaโ
I am able to extract the logs, but I need some pointer where the databricks spend a longer time during job execution.
โ01-31-2022 12:36 AM
Hi @Rajesh Kannan Rโ , This article describes how to use monitoring dashboards to find performance bottlenecks in Spark jobs on Azure Databricks.
โ04-07-2024 12:53 PM
Hi @Kaniz ,
I am saving logs of my databricks Job Compute From ADF, How can i open those files that present in dbfs location.
Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections.
Click here to register and join today!
Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.