12-07-2021 10:48 PM
I'm running a scheduled job on Job clusters. I didnt mention the log location for the cluster. Where can we get the stored logs location.
Yes, I can see the logs in the runs, but i need the logs location.
12-08-2021 06:28 AM
Hi @karthick J ,
You may get more information over here:-
https://docs.databricks.com/clusters/configure.html#cluster-log-delivery
12-07-2021 11:49 PM
Hi @ kjoth! Thanks for your question! Let's see if your peers in the community have an answer to your question first. Or else I will get back to you soon. Thanks.
12-08-2021 12:57 AM
I do not know the default location of the logs.
But it is very easy to define where the logs should be shipped:
https://docs.microsoft.com/en-us/azure/databricks/clusters/configure#cluster-log-delivery
Like that you control the location.
12-08-2021 02:22 AM
Yes, I'm aware of the option of providing log location. Ran some jobs without configuring the log location. Need to know the default location, for exporting the logs.
12-08-2021 06:28 AM
Hi @karthick J ,
You may get more information over here:-
https://docs.databricks.com/clusters/configure.html#cluster-log-delivery
09-22-2022 05:03 AM
Hi @karthick J Did you get this?
09-22-2022 05:05 AM
@Kaniz Fatma
I am unable find where the logs were stored defaultly without giving any path in advanced options.Would Please help me with this?
09-25-2022 09:54 PM
Hi @Sai Kalyani P , Yes it helped. Thanks
09-26-2022 03:11 AM
@karthick J
would you please help me to find out the location of the logs location?I was unable to find out that one
09-26-2022 07:52 AM
Hi @Sai Kalyani P ,
In the databricks one of the cluster run this command to get the spark configs
spark.sparkContext.getConf().getAll()
The above command will list all configs.
In this returned result, search for this config
('spark.databricks.eventLog.dir', 'eventlogs') This is the place where eventlogs are stored.
Alos check this property, where its by default disabled.
('spark.eventLog.enabled', 'false'),
Check these two files.
ls -ll /databricks/driver/logs/stdout
ls -ll /databricks/spark/logs/
By default these logs are not saved regularly.
We need to mention the cluster log location if we want it ti be stored reliably.
09-26-2022 09:48 AM
@karthick J
Thank you!!
Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections.
Click here to register and join today!
Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.