live spark driver log analysis

[Data_Engineer3]

In databricks, if we want to see the live log of the exuction we can able to see it from the driver log page of the cluster.

But in that we can't able to search by key word instead of that we need to download every one hour log file and live logs are updating as frequent even auto fetch is enabled, we have to referesh the page frequent. So is there any possibility to integrate the driver logs with any log monitoring tool for search and update the live log, or is there any possible to do that?

[shan_chandra]

@Data_Engineer3  - You can configure a log delivery location for the cluster. Both worker and driver logs are delivered to the location you specify. Also, you can check to see if you can integrate with Splunk for log monitoring. 

[Data_Engineer3]

Hi @shan_chandra ,

It is like we are putting our driver log into another cloud platform, But here I want to check the live log in local machine tools, is this possible? 

[shan_chandra]

@Data_Engineer3  - Live log analysis in local machine is not possible. we can do a deferred analysis by copying the logs from the log delivery location(use filestore location) to local. 

Please refer to the below post on how to copy the logs from dbfs to local.

https://community.databricks.com/t5/data-engineering/how-to-download-a-file-from-dbfs-to-my-local-co...