cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Where is the cluster logs of the Databricks Jobs stored.

kjoth
Contributor II

I'm running a scheduled job on Job clusters. I didnt mention the log location for the cluster. Where can we get the stored logs location.

Yes, I can see the logs in the runs, but i need the logs location.

1 ACCEPTED SOLUTION

Accepted Solutions

Kaniz
Community Manager
Community Manager

Hi @karthick J​ ,

  1. You need to check the following property in the configuration file.

  • You can access the location wherein you'll get the logs.

You may get more information over here:-

https://docs.databricks.com/clusters/configure.html#cluster-log-delivery

Screenshot 2021-12-08 at 7.21.37 PM 

View solution in original post

10 REPLIES 10

Kaniz
Community Manager
Community Manager

Hi @ kjoth! Thanks for your question! Let's see if your peers in the community have an answer to your question first. Or else I will get back to you soon. Thanks.

-werners-
Esteemed Contributor III

I do not know the default location of the logs.

But it is very easy to define where the logs should be shipped:

https://docs.microsoft.com/en-us/azure/databricks/clusters/configure#cluster-log-delivery

Like that you control the location.

Yes, I'm aware of the option of providing log location. Ran some jobs without configuring the log location. Need to know the default location, for exporting the logs.

Kaniz
Community Manager
Community Manager

Hi @karthick J​ ,

  1. You need to check the following property in the configuration file.

  • You can access the location wherein you'll get the logs.

You may get more information over here:-

https://docs.databricks.com/clusters/configure.html#cluster-log-delivery

Screenshot 2021-12-08 at 7.21.37 PM 

kalyani1
New Contributor III

Hi @karthick J​ Did you get this?

kalyani1
New Contributor III

@Kaniz Fatma​ 

I am unable find where the logs were stored defaultly without giving any path in advanced options.Would Please help me with this?

kjoth
Contributor II

Hi @Sai Kalyani P​ , Yes it helped. Thanks

kalyani1
New Contributor III

@karthick J​ 

would you please help me to find out the location of the logs location?I was unable to find out that one

Hi @Sai Kalyani P​ ,

In the databricks one of the cluster run this command to get the spark configs

spark.sparkContext.getConf().getAll() 

The above command will list all configs.

In this returned result, search for this config

 ('spark.databricks.eventLog.dir', 'eventlogs') This is the place where eventlogs are stored.

Alos check this property, where its by default disabled.

 ('spark.eventLog.enabled', 'false'),

Check these two files.

ls -ll /databricks/driver/logs/stdout

ls -ll /databricks/spark/logs/

By default these logs are not saved regularly.

We need to mention the cluster log location if we want it ti be stored reliably.

kalyani1
New Contributor III

@karthick J​ 

Thank you!!

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.