Where is the cluster logs of the Databricks Jobs stored.

[kjoth]

I'm running a scheduled job on Job clusters. I didnt mention the log location for the cluster. Where can we get the stored logs location.

Yes, I can see the logs in the runs, but i need the logs location.

[-werners-]

I do not know the default location of the logs.

But it is very easy to define where the logs should be shipped:

https://docs.microsoft.com/en-us/azure/databricks/clusters/configure#cluster-log-delivery

Like that you control the location.

[kjoth]

Yes, I'm aware of the option of providing log location. Ran some jobs without configuring the log location. Need to know the default location, for exporting the logs.

[kalyani1]

Hi @karthick J​ Did you get this?

[kalyani1]

@Kaniz Fatma​ 

I am unable find where the logs were stored defaultly without giving any path in advanced options.Would Please help me with this?

[kjoth]

Hi @Sai Kalyani P​ , Yes it helped. Thanks

[kalyani1]

@karthick J​ 

would you please help me to find out the location of the logs location?I was unable to find out that one

[kjoth]

Hi @Sai Kalyani P​ ,

In the databricks one of the cluster run this command to get the spark configs

spark.sparkContext.getConf().getAll() 

The above command will list all configs.

In this returned result, search for this config

 ('spark.databricks.eventLog.dir', 'eventlogs') This is the place where eventlogs are stored.

Alos check this property, where its by default disabled.

 ('spark.eventLog.enabled', 'false'),

Check these two files.

ls -ll /databricks/driver/logs/stdout

ls -ll /databricks/spark/logs/

By default these logs are not saved regularly.

We need to mention the cluster log location if we want it ti be stored reliably.

[kalyani1]

@karthick J​ 

Thank you!!