Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-26-2022 07:52 AM
Hi @Sai Kalyani P ,
In the databricks one of the cluster run this command to get the spark configs
spark.sparkContext.getConf().getAll()
The above command will list all configs.
In this returned result, search for this config
('spark.databricks.eventLog.dir', 'eventlogs') This is the place where eventlogs are stored.
Alos check this property, where its by default disabled.
('spark.eventLog.enabled', 'false'),
Check these two files.
ls -ll /databricks/driver/logs/stdout
ls -ll /databricks/spark/logs/
By default these logs are not saved regularly.
We need to mention the cluster log location if we want it ti be stored reliably.