live spark driver log analysis
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-26-2024 08:46 AM
In databricks, if we want to see the live log of the exuction we can able to see it from the driver log page of the cluster.
But in that we can't able to search by key word instead of that we need to download every one hour log file and live logs are updating as frequent even auto fetch is enabled, we have to referesh the page frequent. So is there any possibility to integrate the driver logs with any log monitoring tool for search and update the live log, or is there any possible to do that?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-27-2024 08:59 AM
@Data_Engineer3 - You can configure a log delivery location for the cluster. Both worker and driver logs are delivered to the location you specify. Also, you can check to see if you can integrate with Splunk for log monitoring.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-28-2024 02:35 AM
Hi @shan_chandra ,
It is like we are putting our driver log into another cloud platform, But here I want to check the live log in local machine tools, is this possible?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-07-2024 08:47 AM
@Data_Engineer3 - Live log analysis in local machine is not possible. we can do a deferred analysis by copying the logs from the log delivery location(use filestore location) to local.
Please refer to the below post on how to copy the logs from dbfs to local.
