cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

live spark driver log analysis

Data_Engineer3
Contributor III

In databricks, if we want to see the live log of the exuction we can able to see it from the driver log page of the cluster.

But in that we can't able to search by key word instead of that we need to download every one hour log file and live logs are updating as frequent even auto fetch is enabled, we have to referesh the page frequent. So is there any possibility to integrate the driver logs with any log monitoring tool for search and update the live log, or is there any possible to do that?

3 REPLIES 3

shan_chandra
Esteemed Contributor

@Data_Engineer3  - You can configure a log delivery location for the cluster. Both worker and driver logs are delivered to the location you specify. Also, you can check to see if you can integrate with Splunk for log monitoring. 

Data_Engineer3
Contributor III

Hi @shan_chandra ,

It is like we are putting our driver log into another cloud platform, But here I want to check the live log in local machine tools, is this possible? 

@Data_Engineer3  - Live log analysis in local machine is not possible. we can do a deferred analysis by copying the logs from the log delivery location(use filestore location) to local. 

Please refer to the below post on how to copy the logs from dbfs to local.

https://community.databricks.com/t5/data-engineering/how-to-download-a-file-from-dbfs-to-my-local-co...

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group