cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Download event, driver, and executor logs

RajeshRK
Contributor

Hi Team,

I can see logs in Databricks console by navigating workflow -> job name -> logs. These logs are very generic like stdout, stderr and log4-avtive.log.

How to download event, driver, and executor logs at once for a job?

Regards,

Rajesh.

1 ACCEPTED SOLUTION

Accepted Solutions

RajeshRK
Contributor

@Kaniz Fatma​ @John Lourdu​ @Vidula Khanna​ 

Hi Team,

I managed to download logs using the Databricks command line as below:

  1. Installed the Databricks command line on my Desktop (pip install databricks-cli)
  2. Configured the Databricks cluster URL and personal token. (databricks configure --token)
  3. Followed the below procedure to download the logs

1. Get the job run id using below command
# databricks runs list | grep -i running
 
2. Identify the cluster id using the run id
# databricks clusters list | grep <run id from the above command>
 
3. Copied the logs from Databricks cluster to my local desktop
# databricks fs cp -r <databricks log location/<cluster id got from the above command>  <location in my desktop>

Regards,

Rajesh.

View solution in original post

7 REPLIES 7

Anonymous
Not applicable

@Rajesh Kannan R​ :

To download event, driver, and executor logs at once for a job in Databricks, you can follow these steps:

  1. Navigate to the "Jobs" section of the Databricks workspace.
  2. Click on the job name for which you want to download logs.
  3. Click on the "Logs" tab to view the logs for the job.
  4. Scroll down to the "Log Storage" section and click on the "Download Logs" button.
  5. In the "Download Logs" dialog box, select the logs you want to download, such as "Driver Logs", "Executor Logs", and "Event Logs".
  6. Specify the time range for the logs and the format in which you want to download them.
  7. Click on the "Download" button to download the logs in a ZIP file.

Note that the download logs feature is available in Databricks Enterprise Edition and Databricks Community Edition. However, the specific log files available for download may vary depending on the Databricks cluster configuration and the permissions of the user running the job.

Hi Teja,

Thank you for replying.

From Databricks Workspace

1) First, I navigated to Workflows -> Jobs and then searched for the job

2) Opened the job

3) Clicked the “Logs” and then directed to “Spark Driver Logs”.

4) There is no option for "Log Storage". I have attached the screenshot.

Regards,

Rajesh.

Anonymous
Not applicable

Hi @Rajesh Kannan R​ 

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. 

We'd love to hear from you.

Thanks!

Anonymous
Not applicable

@Rajesh Kannan R​ You can configure cluster log delivery in job or interactive clusters. You can select DBFS or S3/ADLS as the destination. Once configured all logs including drive, executor logs, eventlog will be delivered to the destination. You can use replay the sparkUI using eventlog if the sparkui is not loading for the job after cluster is terminated.

https://docs.databricks.com/archive/compute/configure.html#cluster-log-delivery-1

https://kb.databricks.com/clusters/replay-cluster-spark-events?from_search=113068791

@John Lourdu​ @Kaniz Fatma​ @Vidula Khanna​ 

Hi Team,

We use job cluster, and logs default to file system DBFS. The cluster is terminated immediately after the job execution. Are there any ways to download the logs from DBFS from the terminated cluster?

I am thinking of addressing it by using the below options:

  1. Not to terminate the cluster immediately
  2. copy the logs from DBFS to ADLS
  3. terminate the cluster.

Regards,

Rajesh.

Kaniz
Community Manager
Community Manager

Hi @Rajesh Kannan R​ ​​, We haven't heard from you since the last response from @John Lourdu​​, and I was checking back to see if her suggestions helped you.

Or else, If you have any solution, please share it with the community, as it can be helpful to others. 

Also, Please don't forget to click on the "Select As Best" button whenever the information provided helps resolve your question.

RajeshRK
Contributor

@Kaniz Fatma​ @John Lourdu​ @Vidula Khanna​ 

Hi Team,

I managed to download logs using the Databricks command line as below:

  1. Installed the Databricks command line on my Desktop (pip install databricks-cli)
  2. Configured the Databricks cluster URL and personal token. (databricks configure --token)
  3. Followed the below procedure to download the logs

1. Get the job run id using below command
# databricks runs list | grep -i running
 
2. Identify the cluster id using the run id
# databricks clusters list | grep <run id from the above command>
 
3. Copied the logs from Databricks cluster to my local desktop
# databricks fs cp -r <databricks log location/<cluster id got from the above command>  <location in my desktop>

Regards,

Rajesh.

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.