topic Re: live spark driver log analysis in Data Engineering

live spark driver log analysis

Data_Engineer3 — Mon, 26 Feb 2024 16:46:33 GMT

In databricks, if we want to see the live log of the exuction we can able to see it from the driver log page of the cluster.

But in that we can't able to search by key word instead of that we need to download every one hour log file and live logs are updating as frequent even auto fetch is enabled, we have to referesh the page frequent. So is there any possibility to integrate the driver logs with any log monitoring tool for search and update the live log, or is there any possible to do that?

Re: live spark driver log analysis

shan_chandra — Tue, 27 Feb 2024 16:59:23 GMT

@Data_Engineer3 - You can configure a log delivery location for the cluster. Both worker and driver logs are delivered to the location you specify. Also, you can check to see if you can integrate with Splunk for log monitoring.

Re: live spark driver log analysis

Data_Engineer3 — Wed, 28 Feb 2024 10:35:01 GMT

Hi @shan_chandra ,

It is like we are putting our driver log into another cloud platform, But here I want to check the live log in local machine tools, is this possible?

Re: live spark driver log analysis

shan_chandra — Thu, 07 Mar 2024 16:47:37 GMT

@Data_Engineer3 - Live log analysis in local machine is not possible. we can do a deferred analysis by copying the logs from the log delivery location(use filestore location) to local.

Please refer to the below post on how to copy the logs from dbfs to local.

https://community.databricks.com/t5/data-engineering/how-to-download-a-file-from-dbfs-to-my-local-computer-filesystem/td-p/29161