topic Re: Where is the cluster logs of the Databricks Jobs stored. in Data Engineering

Where is the cluster logs of the Databricks Jobs stored.

kjoth — Wed, 08 Dec 2021 06:48:07 GMT

I'm running a scheduled job on Job clusters. I didnt mention the log location for the cluster. Where can we get the stored logs location.

Yes, I can see the logs in the runs, but i need the logs location.

Re: Where is the cluster logs of the Databricks Jobs stored.

-werners- — Wed, 08 Dec 2021 08:57:11 GMT

I do not know the default location of the logs.

But it is very easy to define where the logs should be shipped:

https://docs.microsoft.com/en-us/azure/databricks/clusters/configure#cluster-log-delivery

Like that you control the location.

Re: Where is the cluster logs of the Databricks Jobs stored.

kjoth — Wed, 08 Dec 2021 10:22:37 GMT

Yes, I'm aware of the option of providing log location. Ran some jobs without configuring the log location. Need to know the default location, for exporting the logs.

Re: Where is the cluster logs of the Databricks Jobs stored.

kalyani1 — Thu, 22 Sep 2022 12:03:52 GMT

Hi @karthick J Did you get this?

Re: Where is the cluster logs of the Databricks Jobs stored.

kalyani1 — Thu, 22 Sep 2022 12:05:13 GMT

@Kaniz Fatma

I am unable find where the logs were stored defaultly without giving any path in advanced options.Would Please help me with this?

Re: Where is the cluster logs of the Databricks Jobs stored.

kjoth — Mon, 26 Sep 2022 04:54:06 GMT

Hi @Sai Kalyani P , Yes it helped. Thanks

Re: Where is the cluster logs of the Databricks Jobs stored.

kalyani1 — Mon, 26 Sep 2022 10:11:57 GMT

@karthick J

would you please help me to find out the location of the logs location?I was unable to find out that one

Re: Where is the cluster logs of the Databricks Jobs stored.

kjoth — Mon, 26 Sep 2022 14:52:18 GMT

Hi @Sai Kalyani P ,

In the databricks one of the cluster run this command to get the spark configs

spark.sparkContext.getConf().getAll()

The above command will list all configs.

In this returned result, search for this config

('spark.databricks.eventLog.dir', 'eventlogs') This is the place where eventlogs are stored.

Alos check this property, where its by default disabled.

('spark.eventLog.enabled', 'false'),

Check these two files.

ls -ll /databricks/driver/logs/stdout

ls -ll /databricks/spark/logs/

By default these logs are not saved regularly.

We need to mention the cluster log location if we want it ti be stored reliably.

Re: Where is the cluster logs of the Databricks Jobs stored.

kalyani1 — Mon, 26 Sep 2022 16:48:47 GMT

@karthick J

Thank you!!